What is the best python framework to use in production for running asynchronous process (or threads). Here is my use case : I want to poll an external queue (SQS). Based on the message perform some high I/O intensive tasks. The execution time for tasks is high - roughly 5-10 min poer task.

[edit] Should I write my custom code for the thread / process to wake up every 30 sec and long poll queue to see if there is a message. Is there a library that already takes care of the nuances - heart beat, exception handling etc. I want it to be as simple as Django - I should be able to put in the number of process (or threads) that I want to execute and the timer after which they should wake up

CodePudding user response：

Asyncio is a good choice if you want to use a built-in Python library and you don't need the advanced features provided by other frameworks.

CodePudding user response：

The asyncio library is good but you can also take a look at native concurrent.futures library in Python.

I would also mention that while concurrent.futures can be effective for parallelizing, it may not be the best choice if you have a large number of short-running tasks, as the overhead of starting and managing multiple processes or threads can be significant in this case.

From what you mentioned I assume those 5-10 minute tasks are not great in number in any case so this should not be a concern. But in such cases, you may want to consider using asyncio or a different concurrency library, such as gevent.