I have the function Test below
import concurrent.futures
from multiprocessing import Pool
def Test(n):
print(n)
which i want to run parallelly so I do this
li = ['a','b', 'c']
with concurrent.futures.ProcessPoolExecutor() as executor:
executor.map(Test, li )
but I do not see the output i.e. printing a, b , c
I am running it on Jupyter notebook. not sure if that is the issue. I do not see any output. Although if i call test function individually it runs fine.
I am running it on Windows OS
CodePudding user response:
I would normally issue a close vote on this as a duplicate of this question, but now that we have finally established that you are running under Windows, you have two issues with your code.
- You need to import your worker function
Test
from an external file to use multiprocessing with Jupyter Notebook successfully. But multiprocessing code is best not run from Jupyter Notebook because of this complication and also because terminal output from your subprocesses that would normally be captured and written below the cell is instead written to the Jupyter Notebook logging window (see below). - On platforms that launch new processes using the OS spawn method, such as Windows, you must enclose any code that creates new processes within a
if __name__ == '__main__':
block. In fact, you should place any code at global scope that is not required by your worker function in such a block for improved efficiency as it would otherwise be needlessly executed by every process in your pool as part of its initialization.
File test.py in same directory as your .ipynb file
def Test(n):
print(n)
And your cell
if __name__ == '__main__':
from test import Test
import concurrent.futures
li = ['a','b', 'c']
with concurrent.futures.ProcessPoolExecutor() as executor:
executor.map(Test, li )
Here is the output from the Jupyter Notebook logging window (here I am actually usually using Jupyter Lab, which is built on top of Jupyter Notebook):
[I 06:08:25.599 LabApp] JupyterLab extension loaded from c:\program files\python38\lib\site-packages\jupyterlab
[I 06:08:25.600 LabApp] JupyterLab application directory is c:\program files\python38\share\jupyter\lab
[I 06:08:25.702 LabApp] Serving notebooks from local directory: C:\Booboo\test
[I 06:08:25.703 LabApp] Jupyter Notebook 6.1.5 is running at:
[I 06:08:25.706 LabApp] http://localhost:8888/?token=156f2a52135726b662215b349b4047dfea557c17d1acb366
[I 06:08:25.706 LabApp] or http://127.0.0.1:8888/?token=156f2a52135726b662215b349b4047dfea557c17d1acb366
[I 06:08:25.707 LabApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 06:08:26.082 LabApp]
To access the notebook, open this file in a browser:
file:///C:/Users/Booboo/AppData/Roaming/jupyter/runtime/nbserver-13472-open.html
Or copy and paste one of these URLs:
http://localhost:8888/?token=156f2a52135726b662215b349b4047dfea557c17d1acb366
or http://127.0.0.1:8888/?token=156f2a52135726b662215b349b4047dfea557c17d1acb366
[I 06:08:39.827 LabApp] Build is up to date
[I 06:08:58.745 LabApp] Kernel started: bac1f587-2c31-45e5-b1ca-7f56dd1929ba, name: python3
a
c
b
If Test
were instead:
def Test(n):
return n * n
And your cell:
if __name__ == '__main__':
from test import Test
import concurrent.futures
li = [1, 2, 3]
with concurrent.futures.ProcessPoolExecutor() as executor:
print(list(executor.map(Test, li)))
Then the printing would be done by your main process and you would see the output beneath the cell.
You should also look at PEP 8 – Style Guide for Python Code when you have a chance. Function names are usually not started with uppercase letters.