how to write very long string to a gzip file in java-CodePudding

i have a very long string, and want to wirt to a gzip file

i try use GZIPOutputStream to write a gzip file

but where has exception when i use string.getBytes()

java.lang.OutOfMemoryError: Requested array size exceeds VM limit
        at java.lang.StringCoding.encode(StringCoding.java:350)
        at java.lang.String.getBytes(String.java:941)

there is my code, what should i do that can write file successfully?

public static void way1() throws IOException {
    String filePath = "foo";
    String content = "very large string";
    try (OutputStream os = Files.newOutputStream(Paths.get(filePath));
         GZIPOutputStream gos = new GZIPOutputStream(os)) {
        gos.write(content.getBytes(StandardCharsets.UTF_8));
    }
}

public static void way2() throws IOException {
    String filePath = "foo";
    String content = "very large string";
    try (OutputStream os = Files.newOutputStream(Paths.get(filePath));
         GZIPOutputStream gos = new GZIPOutputStream(os);
         WritableByteChannel fc = Channels.newChannel(gos)) {
        fc.write(ByteBuffer.wrap(content.getBytes(StandardCharsets.UTF_8)));
    }
}

CodePudding user response：

If you have ResultSet then try something like:

public static void string2Zipfile(ResultSet rs, int columnIndex, Path outputFile) throws SQLException, IOException {
    try (InputStream os = rs.getBinaryStream(columnIndex)) {
        try (GZIPOutputStream gos = new GZIPOutputStream(Files.newOutputStream(outputFile))) {
            os.transferTo(gos);
        }
    }
}

CodePudding user response：

It seems that when you convert String to byte[] (using content.getBytes(StandardCharsets.UTF_8)) it just needs a lot of memory for the byte[]. Instead of the conversion of the full String to byte[] at once create a ByteBuffer from it using the selcted encoding, and then write this ByteBuffer to the GZIPOutputStream, this way you will lower the needed size of memory at least by half. To create the ByteBuffer you can use:

Charset charset = StandardCharsets.UTF_8; 
String content = "very large string";
ByteBuffer  byteBuffer = charset.encode(content );

API of ByteBuffer: https://docs.oracle.com/javase/7/docs/api/java/nio/ByteBuffer.html And this might be usefull: How to put the content of a ByteBuffer into an OutputStream?

Alternativelly you can also increase the amount of memory for the java heap: Increase heap size in Java

All together would be very similar to your way2, smthg like this (I didn't test it)

public static void way2() throws IOException {
    String filePath = "foo";
    String content = "very large string";
    try (OutputStream os = Files.newOutputStream(Paths.get(filePath));
         GZIPOutputStream gos = new GZIPOutputStream(os);
         WritableByteChannel fc = Channels.newChannel(gos)) {
        Charset charset = StandardCharsets.UTF_8; 
       
        ByteBuffer  byteBuffer = charset.encode(content );
        fc.write(byteBuffer );
    }
}