I need to create a large image (w <= 20k, h <= 100k) I divide it into fragments (w = 20k, h = 5k) and store them in the database in a loop. But I still get an out of memory error. in the heap 1.5 gigabytes. Why is there a memory leak? According to my plan, only the current fragment, which weighs about 300 megabytes, should be stored in the heap.

@Transactional
public long createNewChart (int width, int height) {
    Chartographer chartographer = new Chartographer(width, height);
    chartRepo.save(chartographer);

    int number = 0;
    while (height >= 5000) {
        BufferedImage fragment = new BufferedImage(width, 5000, BufferedImage.TYPE_INT_RGB);
        fragmentRepo.save(new Fragment(imageOperator.imageAsBytes(fragment), number, chartographer));
        height -= 5000;
        number  ;
    }
    if (height > 0) {
        BufferedImage fragment = new BufferedImage(width, height % 5000, BufferedImage.TYPE_INT_RGB);
        fragmentRepo.save(new Fragment(imageOperator.imageAsBytes(fragment), number, chartographer));
    }
    return chartographer.getCharId();
}

public byte[] imageAsBytes (BufferedImage image) {
    try(ByteArrayOutputStream stream = new ByteArrayOutputStream()) {
        ImageIO.write(image, "bmp", stream);
        return stream.toByteArray();
    } catch (IOException ex) {
        throw new RuntimeException(ex);
    }
}

CodePudding user response：

If I am not mistaking, according to your code, you are creating a new byte[] array with width, height % 5000 for every save. So the memory is allocated either way, and the garbage collector did not have a chance to remove the older byte[] arrays before out of memory.

You could try to create a local variable byte[] array and attempt to stream.toByteArray() in it, before doing fragementRepo.save() that way java uses the same memory reference and overwrites.

CodePudding user response：

IMHO you are underestimating your heap space requirement.

• For 20_000 by 5_000 pixels the BufferedImage needs at least 400 MB (100_000_000 pixels at 4 bytes per pixel)
• ByteArrayOutputStream starts with a buffer size of 32 bytes and increases the buffer size every time when there is not enough room for writing data, at least doubling the size of the buffer every time
  • in my test the final buffer size in the ByteArrayOutputStream was 536_870_912
  • because that was size was probably reached by doubling the previous size, it temporarily needed 1.5 times that memory
• the final byte array another 300_000_054 bytes to store the final byte array.

The critical point in memory consumption is the line return stream.toByteArray();:

• the BufferedImage cannot be garbage collected (400MB)
• the ByteArrayOutputStream contains a buffer of about 540MB
• the memory for the final byte array needs to be allocated (another 300MB)
• giving a total needed memory at that specific point of 1240MB for the image processing alone (not taking into account all the memory that the rest of your application consumes)

You can somewhat reduce the needed memory by presizing the ByteArrayOutputStream (by about 240MB):

public byte[] imageAsBytes (BufferedImage image) {
    int imageSize = image.getWidth()*image.getHeight()*3   54; // 54 bytes for the BMP header
    try (ByteArrayOutputStream stream = new ByteArrayOutputStream(imageSize)) {
        ImageIO.write(image, "bmp", stream);
        return stream.toByteArray();
    } catch (IOException ex) {
        throw new RuntimeException(ex);
    }
}