C# How to do on-the-fly compression-and-send in a API controller to avoid large memory buffers-CodePudding

What I am trying to achive is to send a backup (stream) from the API to the requestor, and in the process avoid taking up large amounts of RAM. I have been experimenting using Pipes, but didnt get it to work (probably lack of knowledge ;) )

Take a look at the following routine:

 [HttpGet]
        [Route("{projectIdentifier}")]
        [SwaggerResponse(StatusCodes.Status200OK)]
        [SwaggerResponse(StatusCodes.Status404NotFound)]
        public async Task<IActionResult> BackupProject(string projectIdentifier, bool zipped = false)
        {
            if (!TryGetProject(projectIdentifier, out var project))
                return this.NotFoundResult(nameof(projectIdentifier), projectIdentifier);

            string filenamePartName = $"Project {project.Identifier}";
            string saveFilename = Path.GetInvalidFileNameChars().Aggregate(filenamePartName, (current, c) => current.Replace(c, '_'));

            MemoryStream backupstream = new();
            await Core.Iapetus.Instance.Projects.BackupAsync(backupstream, project.ID, zipped); 
            backupstream.Position = 0;
            var file = zipped ? File(backupstream, "application/zip", $"{saveFilename}.json.gz") : File(backupstream, "application/json", $"{saveFilename}.json");
            return file;
        }

The BackupAsync routine supports compression using the GZipSTream class, and this works on-the fly. Unfortunately the compressed (or not) result still lands in the backupstream memory buffer taking up way much more space than I like in some cases.

The question

Is there a way to "skip" the memory buffer, and send the data directly from the (compressed) stream to the File response? And -off course- keeping the whole shabang async

TIA

CodePudding user response：

There isn't a streaming file result type in ASP.NET, but they do provide the support to define your own.

You can use a streaming file result type similar to the one on my blog, updated for ASP.NET 6:

public sealed class FileCallbackResult : FileResult
{
    private readonly Func<Stream, ActionContext, Task> _callback;

    public FileCallbackResult(MediaTypeHeaderValue contentType, Func<Stream, ActionContext, Task> callback)
        : base(contentType.ToString())
    {
        _ = callback ?? throw new ArgumentNullException(nameof(callback));
        _callback = callback;
    }

    public override Task ExecuteResultAsync(ActionContext context)
    {
        if (context == null)
            throw new ArgumentNullException(nameof(context));
        var executor = new FileCallbackResultExecutor(context.HttpContext.RequestServices.GetRequiredService<ILoggerFactory>());
        return executor.ExecuteAsync(context, this);
    }

    private sealed class FileCallbackResultExecutor : FileResultExecutorBase
    {
        public FileCallbackResultExecutor(ILoggerFactory loggerFactory)
            : base(CreateLogger<FileCallbackResultExecutor>(loggerFactory))
        {
        }

        public Task ExecuteAsync(ActionContext context, FileCallbackResult result)
        {
            SetHeadersAndLog(context, result, fileLength: null, enableRangeProcessing: false);
            return result._callback(context.HttpContext.Response.Body, context);
        }
    }
}

It's used like this:

[HttpGet]
[Route("{projectIdentifier}")]
[SwaggerResponse(StatusCodes.Status200OK)]
[SwaggerResponse(StatusCodes.Status404NotFound)]
public IActionResult BackupProject(string projectIdentifier, bool zipped = false)
{
  if (!TryGetProject(projectIdentifier, out var project))
    return this.NotFoundResult(nameof(projectIdentifier), projectIdentifier);

  string filenamePartName = $"Project {project.Identifier}";
  string saveFilename = Path.GetInvalidFileNameChars().Aggregate(filenamePartName, (current, c) => current.Replace(c, '_'));

  return new FileCallbackResult(
      new MediaTypeHeaderValue(zipped ? "application/zip" : "application/json"),
      async (outputStream, _) =>
      {
        await Core.Iapetus.Instance.Projects.BackupAsync(outputStream, project.ID, zipped);
      })
  {
    FileDownloadName = zipped ? $"{saveFilename}.json.gz" : $"{saveFilename}.json",
  };
}

A word of warning (also noted on my blog): error handling is now more awkward. Previously, if BackupAsync threw an exception, you would return an error code to the client. Now that you're streaming the result, the headers (including 200 OK) have already been sent, so if BackupAsync throws an exception, the only way ASP.NET has to notify the client is to just terminate the connection, and different browsers may interpret that differently. I believe most browsers will correctly say "download error" or some such generic message, but it is possible that the user will just end up with a truncated file in that case.