Parse Large Json File ( around 300 Mb ) to list of POJO-CodePudding

I have a large JSON file roughly around 300 MB, I am parsing it using Jackson object mapper:-

private void parseJson(Object obj) {

ObjectMapper map = new ObjectMapper();
Map<String,List<POJOClass>> result = new HashMap<>();
String str = map.writeValueAsString(obj);
map.registerModule(new JSR10module());
result = map.readValue(new ByteInputStream(str.getBytes(StandardCharSets.UTF_8)),
new TypeReference<Map<String,List<POJOClass>>>
() {});
}

Parameter to parseJson is an Object which contains JSON String.

This works fine for JSON files of around 150 MB, however, it starts to fail with Heap Space error when JSON files are around 250-300 MB. I am using jackson 2.4.0

CodePudding user response：

You don't have enough memory in your java process to handle such huge file.

When launching your app use the -xmx option to increase the maximum memory used by your java process:

java -Xmx512m ...

java  -Xmx1G ...

CodePudding user response：

Have you tried streaming over the JSON using, for instance: https://www.baeldung.com/jackson-streaming-api so that you do not have to put everything into memory at once.

CodePudding user response：

For huge big json file, you should use jackson stream api. You can find the document here. It cost little memory than normal way. In general, your POJO must satisfy certain shapes, it should be able to be separated into many independent parts. Such as a list of object. Then with jackson stream api, you can achieve the object in the list one by one.