Run N processes but never reuse the same process-CodePudding

I like to run a bunch of processes concurrently but never want to reuse an already existing process. So, basically once a process is finished I like to create a new one. But at all times the number of processes should not exceed N.

I don't think I can use multiprocessing.Pool for this since it reuses processes.

How can I achieve this?

One solution would be to run N processes and wait until all processed are done. Then repeat the same thing until all tasks are done. This solution is not very good since each process can have very different runtimes.

Here is a naive solution that appears to work fine:

from multiprocessing import Process, Queue
import random
import os
from time import sleep


def f(q):
    print(f"{os.getpid()} Starting")
    sleep(random.choice(range(1, 10)))
    q.put("Done")


def create_proc(q):
    p = Process(target=f, args=(q,))
    p.start()


if __name__ == "__main__":
    q = Queue()

    N = 5
    for n in range(N):
        create_proc(q)

    while True:
        q.get()
        create_proc(q)

CodePudding user response：

Pool can reuse a process a limited number of times, including one time only when you pass maxtasksperchild=1. You might also try initializer to see if you can run the picky once per process parts of your library there instead of in your pool jobs.