I have the following scenario:

• Back end => a geospatial database and the Open Data Cube tool
• API => Users can define parameters (xmin,xmax,ymin,ymax) to make GET requests
• Process => On each requests analytics are calculated and satellite images pixels' values are given back to the user

My question is the following: As the process is quite heavy (it can reserve many GB of RAM) how it is possible to handle multiple requests at the same time? Is there any queue that I can save the requests and serve each one sequently?

Language/frameworks => Python 3.8 and Django

Thanks in advance

CodePudding user response：

Celery Rabbitmq/Redis is probably what you need. In this configuration, your heavy processes become "tasks". When called with .delay() they go in the queue and are not handle by your main process anymore.

You might want to check the tuto

https://docs.celeryproject.org/en/stable/django/first-steps-with-django.html

CodePudding user response：

There are many asynchronous message queueing technologies that allow you to do this, lots of which have Python APIs too.

You probably want to use request-response messaging, to correlate the requests you get with the replies you want to send.

A message queueing technology will allow you to take the requests, store them on a queue, and have your server handle them when it's ready. Storing requests on a queue means that they won't get lost. This also allows your application to scale - as more requests come in, they can be dealt with by multiple application instances and still return only one result!

The answer above recommends celery, which is a great choice for this kind of project. Depending on the requirements you have, you can also use pymqi (example here for using a request-response pattern) ZeroMQ - Multiple Publishers and Listener if you need something more heavy-duty.