Passing function to map or starmap in multiprocessing pool-CodePudding

i've been trying to test this simple piece of code on my machine on Jupyter and the cell just runs indefinitely without outputing anything. Is there some kind of bug or something? I use the exact same piece of code for a pandas process with pool.map and everything just works fine, but can't figure out what is happening here.

import multiprocessing as mp 
pool1 = mp.Pool(processes = 3)

def sumP(a, b):
    return (a * b) / (a - b   1)

f1, f2, f3 = 24, 31, 45
new_rows2 = pool1.starmap(sumP, [(f1, f2), (f2, f3), (f1, f3)]) 
print(new_rows2)

And the cell just keeps running. Doesn't matter if i use pool1.map or pool1.starmap. Do you guys suggest any other way of parallelizing this process?

CodePudding user response：

This happens because the worker processes attempt to import sumP, which also creates another 3 processes, ad-infinitum.

You'll need to put a guard around the multiprocess creation so that workers don't spawn indefinitely:

import multiprocessing as mp 

def sumP(a, b):
    return (a * b) / (a - b   1)

f1, f2, f3 = 24, 31, 45

if __name__ == "__main__":
    pool1 = mp.Pool(processes=3)
    new_rows2 = pool1.starmap(sumP, [(f1, f2), (f2, f3), (f1, f3)]) 
    print(new_rows2)

Look for "Safe importing of main module" under the multiprocessing docs for more info. You should not create a Pool during importing of the main module.