Running multiple processses asynchronously-CodePudding

So I have a main() function which needs to continously start >20 different processes every 5 seconds without waiting them to finish as some might take over 5 minutes to finish. I’ve used threads so far (without joining them) but GIL eventually halts the execution as number of active threads increases. Could I maybe use some sort of async multiprocessing?

CodePudding user response：

You can try to use concurrent.future.ProcessPoolExecutor to asynchronously run functions in multiple processes without overloading your pc:

import time
from concurrent.futures import ProcessPoolExecutor


def background_func(task):
    time.sleep(task)
    return task


with ProcessPoolExecutor(max_workers=10) as pool:
    for task in range(30):
        r = pool.submit(background_func, task) # non-blocking operation
        r.add_done_callback(lambda x: print(f"completed: {x.result()}"))