I have a lambda function that triggers once a file is uploaded to S3. There's a slight possibility that two files might be uploaded at the same time. The thing is, I don't want the lambda function running both files at the same time. I want one file to go first and then another. Is this possible?
CodePudding user response：
Yes, it is possible to have one Lambda function wait for another to finish before triggering. Here are a few different ways you can do this:
Use a lock on the shared resource that both functions need to access. This can be as simple as a file on S3 that each function checks for before proceeding. When one function starts running, it acquires the lock by creating the file. When it finishes, it deletes the file, allowing the other function to acquire the lock and run.
Use the built-in concurrency controls in AWS Lambda. With concurrency controls, you can set the maximum number of concurrent executions for a Lambda function, and the function will automatically queue any additional requests until the number of concurrent executions falls below the limit. This way, if one function is already running, the other will be queued until the first one finishes.
Use AWS Step Functions to create a workflow that orchestrates the execution of your Lambda functions. With Step Functions, you can define a sequence of tasks that run in a specific order, with each task representing a Lambda function. This way, you can ensure that one function runs after another and control the flow of execution.
Each of these approaches has its own trade-offs and considerations, so you'll need to choose the one that best fits your use case. You may also want to consider the complexity of the solution and the amount of additional code or infrastructure it requires.
CodePudding user response：
One way to implement this is to set the concurrency limit for a Lambda function to 1 to ensure that only one instance of the function is active at any given time. Any additional invocations of the function will be throttled until the active instance completes its execution. This will ensure that the function is executed serially, with only one instance running at a time. You can have your S3 bucket send a message to the FIFO queue whenever a new file is uploaded. This approach will ensure that your Lambda function processes the files in the order in which they are added to the queue and will prevent the function from running multiple instances concurrently.
Another way to implement this is to use a distributed lock. When your lambda function starts, it can attempt to acquire the lock by writing a record to the table. This dynamodb write can be transactional write with optimistic locking. If the write operation succeeds, it means that the lambda function has acquired the lock and can proceed with its execution. If the write operation fails, it means that another instance of the lambda function has already acquired the lock. This has a lot of challenges that need to be handled, like how to re-trigger the lambda if the acquisition of lock fails.