How to run a function parallelly in python?-CodePudding

I have the function Test below

import concurrent.futures
from multiprocessing import Pool

def Test(n):
    print(n)

which i want to run parallelly so I do this

li = ['a','b', 'c']
    
with concurrent.futures.ProcessPoolExecutor() as executor:
    executor.map(Test, li )

but I do not see the output i.e. printing a, b , c

I am running it on Jupyter notebook. not sure if that is the issue. I do not see any output. Although if i call test function individually it runs fine.

I am running it on Windows OS

CodePudding user response：

I would normally issue a close vote on this as a duplicate of this question, but now that we have finally established that you are running under Windows, you have two issues with your code.

You need to import your worker function Test from an external file to use multiprocessing with Jupyter Notebook successfully. But multiprocessing code is best not run from Jupyter Notebook because of this complication and also because terminal output from your subprocesses that would normally be captured and written below the cell is instead written to the Jupyter Notebook logging window (see below).
On platforms that launch new processes using the OS spawn method, such as Windows, you must enclose any code that creates new processes within a if __name__ == '__main__': block. In fact, you should place any code at global scope that is not required by your worker function in such a block for improved efficiency as it would otherwise be needlessly executed by every process in your pool as part of its initialization.

File test.py in same directory as your .ipynb file


def Test(n):
    print(n)

And your cell

if __name__ == '__main__':
    from test import Test
    import concurrent.futures


    li = ['a','b', 'c']

    with concurrent.futures.ProcessPoolExecutor() as executor:
        executor.map(Test, li )

Here is the output from the Jupyter Notebook logging window (here I am actually usually using Jupyter Lab, which is built on top of Jupyter Notebook):

[I 06:08:25.599 LabApp] JupyterLab extension loaded from c:\program files\python38\lib\site-packages\jupyterlab
[I 06:08:25.600 LabApp] JupyterLab application directory is c:\program files\python38\share\jupyter\lab
[I 06:08:25.702 LabApp] Serving notebooks from local directory: C:\Booboo\test
[I 06:08:25.703 LabApp] Jupyter Notebook 6.1.5 is running at:
[I 06:08:25.706 LabApp] http://localhost:8888/?token=156f2a52135726b662215b349b4047dfea557c17d1acb366
[I 06:08:25.706 LabApp]  or http://127.0.0.1:8888/?token=156f2a52135726b662215b349b4047dfea557c17d1acb366
[I 06:08:25.707 LabApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 06:08:26.082 LabApp]

    To access the notebook, open this file in a browser:
        file:///C:/Users/Booboo/AppData/Roaming/jupyter/runtime/nbserver-13472-open.html
    Or copy and paste one of these URLs:
        http://localhost:8888/?token=156f2a52135726b662215b349b4047dfea557c17d1acb366
     or http://127.0.0.1:8888/?token=156f2a52135726b662215b349b4047dfea557c17d1acb366
[I 06:08:39.827 LabApp] Build is up to date
[I 06:08:58.745 LabApp] Kernel started: bac1f587-2c31-45e5-b1ca-7f56dd1929ba, name: python3
a
c
b

If Test were instead:

def Test(n):
    return n * n

And your cell:

if __name__ == '__main__':
    from test import Test
    import concurrent.futures


    li = [1, 2, 3]

    with concurrent.futures.ProcessPoolExecutor() as executor:
        print(list(executor.map(Test, li)))

Then the printing would be done by your main process and you would see the output beneath the cell.

You should also look at PEP 8 – Style Guide for Python Code when you have a chance. Function names are usually not started with uppercase letters.