what i wanted to do When the flask application receives a request to to a specific endpoint (eg /scale-up), Since it takes more than 5 minutes to turn on the EC2 instance, I try to run that logic in a different process and send a response directly(don't wait until that process turn on EC2 instance) to the user who sent the request.
But when this happens, I don't know when I can close() and join() the process.
To recap, When a request comes in, it only 'executes' a process that takes a long time, and apart from that I just send a OK response. After that, how can I close and join this process?
I tried to use flask's teardown_request decorator, but this also doesn't seem to be the right way.
Here is my code.
import multiprocessing
from flask import Flask, Response
from multiprocessing import Pool
app = Flask(__name__)
@app.route(f'/scale-up')
def send_scale_up_request():
try:
p = Pool(4)
p.apply_async(scale_up_instance)
logger.info("Executing to restart the instance...")
return Response("Executing to restart the instance...")
# where can I put p.close() and p.join()??
except Exception as e:
logger.error("Error occured")
logger.error(e)
logger.error(traceback.format_exc(limit=None))
def scale_up_instance():
# turn on EC2 instance using aws sdk(boto3)
.....
Thank you very much. This is my first time asking a question here...
CodePudding user response:
There is no reason to be using a multiprocessing pool to be running a single process and so there is also no need for close
and join
. But even running a single Process
, there is really no place or need to put a call to join
for it. The process will run asynchronously and end after your current request has terminated.
Update
To prevent defunct processes because you cannot do a join
on the started process, instead run scale_up_instance
as a thread. If for some reason threading is not sufficient, the thread can start a process that can be joined:
import multiprocessing
from flask import Flask, Response
from threading import Thread
import time
app = Flask(__name__)
@app.route(f'/scale-up')
def send_scale_up_request():
try:
Thread(target=scale_up_instance).start()
logger.info("Executing to restart the instance...")
return Response("Executing to restart the instance...")
except Exception as e:
logger.error("Error occured")
logger.error(e)
logger.error(traceback.format_exc(limit=None))
def scale_up_instance():
# turn on EC2 instance using aws sdk(boto3)
import time
print('starting...')
time.sleep(10)
print('ending.')
# If multiprocessing is required because of GIL restrictions then:
from multiprocessing import Process
p = Process(target=do_scale_up) # defined elsewhere
p.start()
p.join()