Python multiprocessing.Process module not using all CPU cores?-CodePudding

I'm running the following code on windows machine with a powerful inter-core i5 - 12600k processor with 16 cores.

if __name__ == '__main__':
    windows = np.linspace(30,cpu_count()*30,cpu_count()).astype(int)
    significance_level = 0.1
    processes = []
    for window in windows:
        p = Process(target=save_betas_over_window, 
                    args=(assets_historical_returns,
                          factors_historical_returns,
                         ),
                    kwargs={
                        'window':window,
                        'significance_level':0.1
                    }
                   )
        p.start()
        processes.append(p)
    for process in processes:
        process.join()

the code is supposed to run the method save_betas_over_window simultaneously for each of the items in the windows list. I used cpu_count to make the windows list the same length as the number of available CPU-cores on my machine. So, theoretically-as far as I understand-, all the 16 cores should be working. However, when I checked the resource monitor, it shows that only 4 cores [12:15] are used up to 100% while the remaining cores are barely used.

my question is, why some cores are used and others not? isn't it supposed that multiprocessing.Process module assigns processes to available CPU cores that are not used, and if so, why some cores are used up to 100% while others are not used at all?

I'm new to parallel processing and this is so confusing to me, Appreciate any help, thanks in advance.

CodePudding user response：

Use Multiprocessing module in python Pool method of Multiprocessing lets you use the number of CPUs

I will show you an example

p = Pool(multiprocessing.cpu_count() - 2)
p.map(callfunction, val_set)

Here, We have defined the a function named 'callfunction' which takes an argument lets say 'val'

val_set contains data in a set values. Let's say -- {'a', 'b'}

So, here one CPU will call callfunction(a) Other one will call callfunction(b)

According to the code I provided, cpu_count() - 2 will assign a maximum of total cpus - 2 present in the server.

CodePudding user response：

UPDATE:
The issue was solved by