What are the ways of storing data coming from socket every seconds?-CodePudding

I have socket connection in my backend (express.js) and data coming from that socket every seconds, I want to store these values to my database (postgresql). Since data is coming every seconds I do not want to store them when they come because it will be lots of writing operations. So I'm keeping them in tempValues and store the values to the database in every 1 min. But I feel like there has to be better way of doing this instead of using setInterval. Maybe a 3rd party library or something else? I'm okay to loose temporary data if server crashed or something happened.

Here is my current code block:

let tempValues = [];

setInterval(() => {
  tempValues.map(async (option, i) => {
    await pool.query(
      "INSERT INTO plc_options (name, value, date) VALUES ($1, $2, $3)",
      [option.name, option.value, option.timestamp]
    );
    if (i === tempValues.length - 1) tempValues = [];
  })

}, 60000);


socket.onAny((eventName, ...args) => {
  tempValues.push({ name: eventName, value: args[0], timestamp: new Date() })
});

CodePudding user response：

You don't need a 3rd party library. Your idea is generally ok. It is known as buffering. I would improve it a bit though.

let _tempValues = [];

function getTempValues() {
    return _tempValues;
}

function getAndClearTempValues() {
    let oldTempValues = _tempValues;
    _tempValues = [];
    return oldTempValues;
}

setInterval(() => {
  let values = getAndClearTempValues();
  values.map(async (option, i) => {
    await pool.query(
      "INSERT INTO plc_options (name, value, date) VALUES ($1, $2, $3)",
      [option.name, option.value, option.timestamp]
    );
  })

}, 60000);


socket.onAny((eventName, ...args) => {
  let values = getTempValues();   
  values.push({ name: eventName, value: args[0], timestamp: new Date() })
});

The major change is that I swap old tempValues with an empty array before looping through the array. That way we avoid a potential pitfal, when you asynchronously loop through the array, while the other code modifies it. As a bonus you don't need the if statement inside setInterval.