StreamingResponseBody heap usage-CodePudding

i have got simple method in controller which streams content from database, streaming works as intended, download starts right after calling endpoint. Problem is heap usage, streaming 256 MB file takes 1GB heap space. If I would replace service.writeContentToStream(param1, param2, out) with method that reads data from local file to input stream and copying to passed output stream result is same. Biggest file I can stream is 256 MB. Is there possible solution to overcome heap size limit?

    @GetMapping("/{param1}/download-stream")
    public ResponseEntity<StreamingResponseBody> downloadAsStream(
            @PathVariable("param1") String param1,
            @RequestParam(value = "param2") String param2
    ) {
        Metadata metadata = service.getMetadata(param1);
        StreamingResponseBody stream = out ->  service.writeContentToStream(param1, param2, out);
           return ResponseEntity.ok()             
                .header(HttpHeaders.CONTENT_DISPOSITION, "attachment;"   getFileNamePart()   metadata.getFileName())
                .header(HttpHeaders.CONTENT_LENGTH, Long.toString(metadata.getFileSize()))
                .body(stream);
    }

service.writeContentToStream method

 try (FileInputStream fis = new FileInputStream(fileName)) {
     StreamUtils.copy(fis, dataOutputStream);
 } catch (IOException e) {
     log.error("Error writing file to stream",e);
 }

Matadata class contains only information about filesize and filename, there is no content stored there

CodePudding user response：

Some ideas:

Run the server inside the Java profiler. For example JProfiler (it costs money).
Try ServletResponse.setBufferSize(...)
Check, if you have some filters configured in the application.
Check the output buffer of the application server. In case of the Tomcat it could be quite tricky. It has a long list of possible buffers:

https://tomcat.apache.org/tomcat-8.5-doc/config/http.html

CodePudding user response：

I wrote an article back in 2016 regarding StreamingResponseBody when it was first released. You can read that to get more of an idea. But even without that what you are trying to do with the following code is not scalable at all (Imagine 100 users concurrently trying to download).

 try (FileInputStream fis = new FileInputStream(fileName)) {
     StreamUtils.copy(fis, dataOutputStream);
 } catch (IOException e) {
     log.error("Error writing file to stream",e);
 }

The above code is very memory intensive and nodes with high memory can only work with this and you always will have an upper bound on the file size (Can it download a 1TB file in 5 years?)

What you should do is the following;

try (FileInputStream fis = new FileInputStream(fileName)) {
    byte[] data = new byte[2048];
    int read = 0;
    while ((read = fis.read(data)) > 0) {
        dataOutputStream.write(data, 0, read);
    }
    dataOutputStream.flush();
} catch (IOException e) {
    log.error("Error writing file to stream",e);
}

This way your code can download files of any size given that the user is able to wait and will not require a lot of memory