I'm working on a project that involves a large JSON file, basically a multidimensional array dumped in JSON form, but the overall size would be larger than the amount of memory I have. If I load it in as a string and then parse the string, that will consume all of the memory.

Are there any methods to limit the memory consumption, such as only retrieving data between specific indices? Could I implement that using solely the Nlohmann json library/the standard libraries?

CodePudding user response：

RapidJSON and others can do it. Here's an example program using RapidJSON's "SAX" (streaming) API: https://github.com/Tencent/rapidjson/blob/master/example/simplereader/simplereader.cpp

This way, you'll get an event (callback) for each element encountered during parsing. The memory consumption of the parsing itself will be quite small.

CodePudding user response：

Could you please specify the context of your question

• What programming language you are using (NodeJS, Vanilla JavaScript, Java, React)
• What environment your code is running (Monolithic app on a server, AWS Lambda, Serverless)

Computing large JSON files can consume a lot of memory resources on a server, perhaps, make your app to crash. I have experienced first-hand, that manipulating large JSON files on my local computer with 8 GB of memory RAM is not a problem using a NodeJS script to compute the large JSON files payloads. However, trying to run those large JSON payloads in an application running on a server give me problems too.

I hope this helps.