Multiprocessing changed in python 3.9 from 3.7 on MacOSx, how to fix?-CodePudding

Python 3.9 multiprocessing seems to run many extra loops over the whole file, while 3.7 didnt do this. The machine has 8 cores thus 8 loops. How do I fix this. This is not what I am expecting -- was the threading module also reworked, does this problem happen in that module also. This is the code:

 !/usr/bin/python
from   time            import strftime
import multiprocessing as mp

print (strftime("%H:%M:%S")   ' Run line5 ' )

def pullLast():  
    print(12345678998765432345678)
    return 1

def mprocULwork(uplist): # build a queue with tuple
    ULsyms = [e for e in uplist if e is not None] 
    p_size          = mp.cpu_count()
    pool            = mp.Pool(processes=p_size, maxtasksperchild=400,)
    pool_outputs    = pool.map(ULProcess, ULsyms)
    pool.close()    # no more tasks
    pool.join()     # wrap up current tasks
    del pool
    print (strftime("%H:%M:%S"), "Finished ULwork")

def ULProcess(symbol):
    pSet    = 0
    print(pSet,symbol)
    
if __name__ == '__main__': 
    pSet    = 1
    symlist = ["G","Y","S" ]
    ullist  = symlist

    global lastPriceDF
    lastPriceDF  = pullLast()

    mprocULwork(ullist)                ####### <<<<<<<<<<<<<<<<<<main entry
    print (strftime("%H:%M:%S"), "post")
   
print (strftime("%H:%M:%S"),  'Exiting....line last' )

This the output from python 3.7:

10:08:58 Run line5 
12345678998765432345678
0 G
0 Y
0 S
10:08:58 Finished ULwork
10:08:58 post
10:08:58 Exiting....line last

This is the output from 3.9:

10:20:44 Run line5 
12345678998765432345678
10:20:44 Run line5 
10:20:44 Exiting....line last
0 G
0 Y
0 S
10:20:44 Run line5 
10:20:44 Exiting....line last
10:20:44 Run line5 
10:20:44 Exiting....line last
10:20:44 Run line5 
10:20:44 Exiting....line last
10:20:44 Run line5 
10:20:44 Exiting....line last
10:20:44 Run line5 
10:20:44 Exiting....line last
10:20:44 Run line5 
10:20:44 Exiting....line last
10:20:44 Run line5 
10:20:44 Exiting....line last
10:20:44 Finished ULwork
10:20:44 post
10:20:44 Exiting....line last

CodePudding user response：

Psychic debugging: You're on macOS. In 3.8, the multiprocessing module switched from using the 'fork' start method by default to the 'spawn' start method, because the high-level macOS system libraries are not fork-safe and can cause crashes. Part of how the 'spawn' method simulates forking is by reimporting the __main__ module (but with a different __name__ so a properly guarded module won't redo anything that's guarded).

If you want to roll the dice, go ahead and explicitly opt-in to the 'fork' start method with multiprocessing.set_start_method('fork'). More properly, fix your code to make it follow the multiprocessing guidelines for the 'spawn' (and 'forkserver') method.