I am using AWS Lambda to render videos using ffmpeg. It is a basic function that downloads external clips, merges those, then uploads the result to S3.

This function needs to be able to scale quickly because I need to generate a lot of videos in a short time period.

The problem is that I reach the /tmp disk limit of 512MB because the same lambda environment is reused by the different requests. I have increased that memory limit to temporarily solve the problem but it does not feel scalable.

I have looked into EFS but the requirement to use a VPC and a NAT to have internet access sounds really heavy for my simple needs.

Is there a way to output the result of ffmpeg directly in S3 (thus not needing the temporary folder)? Or perhaps force to respawn a clean lambda environment if the memory is about to be full?

CodePudding user response：

See AWS Lambda Now Supports Up to 10 GB Ephemeral Storage:

AWS Lambda now allows you to configure ephemeral storage (/tmp) between 512 MB and 10,240 MB. You can now control the amount of ephemeral storage a function gets for reading or writing data, allowing you to use AWS Lambda for ETL jobs, ML inference, or other data-intensive workloads.

Note that each invocation of your Lambda function can safely remove files that are present in /tmp - you will never have two concurrent Lambda invocations using the same runtime environment. Any files present there are left over from a prior, now complete, invocation of the Lambda function. If there's no need to share files from one Lambda invocation to the next, then your Lambda function can delete those files before exiting to ensure the next invocation has a clean /tmp folder.