I've been tasked to crawl urls and abruptly stop the AWS Lambda after 30 seconds after which the number of URLs crawled will be taken as an evaluation metric. In a simple architecture where I have 1 lambda that takes a file and loops through it and writes to the database, I could have simply asked it to timeout after 30 seconds. For the sake of my learning and to meet the other criteria of scaling, the architecture I adopted is this:

So even if I timeout my lambdas, they will run again given the URLs are being sent as events by fargate. The point of fargate is to be able to download a huge file as Lambda has limitations. The use of events will then help me achieve scale by simply allowing more concurrent lambdas. Can I somehow stop eventbus to freeze sending or receiving notifications after 30 seconds? Can I stop all the computes somehow?

I could send errors etc (I only get timeout errors in the past) to a dead queue or an SNS topic and show resiliency in the system to abrupt crashes. I could also demonstrate the number of URLs crawled by showing logs. But assume these measures do not satisfy the evaluator, is there anything I can do?

I can add delays to messages and queues but how would that do anything? I can't add a delay after a certain time period. That would have worked.

CodePudding user response：

One way would be to disable the Event Rule after a set period of time using the cli or SDK from your container:

$ aws events disable-rule --name MyRule --event-bus-name MyEventBus

Another option to stop all further lambda invocation is to set the concurrency limit to 0 as per How to kill/terminate a running AWS Lambda function? :

$ aws lambda put-function-concurrency --function-name my-function --reserved-concurrent-executions 0

This will not stop the executions that are already running - that cannot be done with Lambda.

Your database can then be filtered by the insertion timestamp to remove all writes that happen after the threshold.