Node.js - Asynchronously edit JSON file and prevent interferance from other async calls-CodePudding

I am making a Discord bot in node.js, and need to store some per-guild data. I want to store it in separate JSON files in data/<guild.id>.json using the built in fs/promises.

I have a function setGuildData(guildID: string, dataToAssign: object) that reads the current JSON file, parses the JSON to fileData, then assigns the new data using Object.assign() to finally stringify the JSON and overwrite the file.

But I am worried that if two async function calls try to edit this JSON file at the same time, they would interfere:

file: {a:0, b:1}
call 1: read the file
call 2: read the file
call 1: edit the object to insert {c:2}
call 2: edit the object to insert {d:3}
call 1: stringify and write {a:0, b:1, c:2}
call 2 stringify and write {a:0, b:1, d:3}
file: {a:0, b:1, d:3}

What I want to happen is that when call1 reads the JSON, it blocks any other read calls for that file until it is finished. This will cause call 2 to read {a:0, b:1, c:2} and add {d:3} to that instead, giving the desired result of {a:0, b:1, c:2, d:3}

So my question is:
How do I open a file, such that I can read it, do stuff with the data, and then overwrite it all whilst blocking other async calls from opening the file at the same time?

The performance impact of blocking this one file for up to a hundred milliseconds isn't bad, because this file is only for that one guild, and I expect to see only a few requests per minute per guild at peak times, and rarely two simultaneous requests. I just want to be on the safe side and not accidentally delete any data like what happened to {c:2} in the example.

I could make a global queue in JS that prevents double file writes, but I would rather have is an OS level way to temporarily block this file. In case this is OS specific, I am using Linux.

Here is what I have now:

const fsPromises = require('node:fs/promises');
const path = require('node:path')

const dataPath = path.join(__dirname, 'data');

async function setGuildData(guildID, newData) {
    let fileHandle = await fsPromises.open(path.join(dataPath, `${guildID}.json`), "w "); // open the file. note 1
    let fileData; // make a variable for the file data
    try {
        fileData = JSON.parse(await fileHandle.readFile()); // try to read and parse the file.
    } catch (err) {
        fileData = {}; // if that doesn't work, assume the file was not made yet and return an empty object
        // yes I tried logging the error, see note 1
    }
    await fileHandle.write(JSON.stringify(Object.assign(fileData, newData)), 0); // use Object.assign to overwrite some properties or create new ones, and write this to the file
    await fileHandle.close(); // close the filehandle.
}

Note 1: The "w " mode seems to clear the file upon opening. I want a mode that allows me to first read and then overwrite the file, all while keeping the file opened to prevent other async calls from interfering.

CodePudding user response：

For Linux there aren't really any ways to block a file like this, but you can use Promises to make an atomic guard. This is a queue system that you can use to queue up any file edit requests.

Below is an example, without using the makeAtomic the answer would be 1, instead of 2.

const sleep = ms => new Promise(r => setTimeout(r, ms));

function makeAtomic() {
    let prom = Promise.resolve(null);
    function run(cb) {
        const oProm = prom;
        prom = (async () => {
            try {
              await oProm;
            } finally {
              return cb();
            }
        })();
        return prom;
    };
    return run;
}

const atom = makeAtomic();

let v = 0;

async function addOne() {
  return atom(async c => {
    let ov = v;
    await sleep(200);
    v = ov   1;
  });
}

async function test() {
  await Promise.all([addOne(), addOne()]);
  console.log(v);
}

test().catch(e => console.error(e));

Note: You can basically share the atomic promise with other parts of code, were serialization is important, it doesn't have to be the same code, handy for places were you want to do multiple operations at the same time, and of course you can have multiple makeAtomics, doing this makes parallel operations very simple and of course avoids race conditions that can easily occur in async coding.