I was required to fetch data from a site and show data in recent 10 minutes time that meets a condition. The first solution that came to my mind was storing in a global variable but it is a bad practice I think. My attempt:

app.get("/report", async (req, res) => {
    const violations = [];
    const xml_string = await fetchXML(drones_url);
    xml2js.parseString(xml_string, async (err, json) => {
        if (err) {
          throw err
        }
        const drones = json['report'].capture['0'].drone

        //traditional for loop to allow the use of await statements
        for(let i = 0; i < drones.length;   i) {
            if(pointIsInNDZ(drones[i].positionX, drones[i].positionY)) {
                //attempt to fetch violator's information 
                let fetch_response = await fetch(pilot_url   drones[i].serialNumber[0]);
                if(fetch_response.ok) {
                    const pilot_json = await fetch_response.json();
                    violations.push(pilot_json);
                }
            };
        }
        res.json(violations);
        
    });

});

This obviously get the current violations but cannot persist it. Any solutions?

CodePudding user response：

If your server will never be clustered (or is not clustered now and you don't need a clustered design) and you want to save data that is available to all future requests (regardless of what user they come from), then you can save that data in a module-level variable or some data structure that is stored in a module-level variable.

You do have to make sure several things are done with relation to that data:

1. Make sure you aren't just growing that data forever and ever. If you add to it regularly, then you will need to be removing old data periodically so it doesn't just take up more and more memory, growing forever.

2. You have to make sure the amount of data you're storing is consistent with the memory available in your environment.

3. You have to make sure the way you store and retrieve the data is safe from concurrency issues. Generally, if you gather data and then insert it into the module level data structure in one synchronous operation, you should be safe. But, if you insert some data, do an asynchronous operation, insert some other data related to the first data, then you're setting yourself up for possible concurrency issues. As always, the details of concurrency issues depend upon the specific data and how it's accessed and stored.

4. If you're collecting the data in a series of asynchronous operations, then you will definitely have possible concurrency issues with another request trying to do the same thing at the same time.

P.S. If your server was clustered or you want a clustered design, then instead of storing the data in your server instance's memory, you would store the data in some sort of centrally accessible data store such as a disk-based database (like mongodb) or it could be a memory-based data store (such as redis). With multiple servers accessing the data, you will have to be even more aware of concurrency issues.