Parallel computing in Python Similar to MATLAB-CodePudding

I have been using parfor in MATLAB to run parallel for loops for quite some time. I need to do something similar in Python but I cannot find any simple solution. This is my code:

t = list(range(1,3,1))
G = list(range(0,3,2))
results = pandas.DataFrame(columns = ['tau', 'p_value','G','t_i'],index=range(0,len(G)*len(t)))
counter = 0  
for iteration_G in list(range(0,len(G))):
    for iteration_t in list(range(0,len(t))):
        
        matrix_1,matrix_2 = bunch of code


        tau, p_value = scipy.stats.kendalltau(matrix_1, matrix_2)
        results['tau'][counter] = tau
        results['p_value'][counter] = p_value
        results['G'][counter] = G[iteration_G]
        results['t_i'][counter] = G[iteration_t]
        counter = counter   1

I would like to use the parfor equivalent in the first loop.

CodePudding user response：

I'm not familiar with parfor, but you can use the joblib package to run functions in parallel.

In this simple example there's a function that prints its argument and we use Parallel to execute it multiple times in parallel with a for-loop

import multiprocessing
from joblib import Parallel, delayed


# function that you want to run in parallel
def foo(i):
    print(i)


# define the number of cores (this is how many processes wil run)
num_cores = multiprocessing.cpu_count()

# execute the function in parallel - `return_list` is a list of the results of the function
# in this case it will just be a list of None's
return_list = Parallel(n_jobs=num_cores)(delayed(foo)(i) for i in range(20))

If this doesn't work for what you want to do, you can try to use numba - it might be a bit more difficult to set-up, but in theory with numba you can just add @njit(parallel=True) as a decorator to your function and numba will try to parallelise it for you.

CodePudding user response：

I found a solution using parfor. It is still a bit more complicated than MATLAB's parfor but it's pretty close to what I am used to.

t = list(range(1,16,1))
G = list(range(0,62,2))
for iteration_t in list(range(0,len(t))):
    @parfor(list(range(0,len(G))))
    def fun(iteration_G):
        result = pandas.DataFrame(columns = ['tau', 'p_value'],index=range(0,1))

        matrix_1,matrix_2 = bunch of code   



        tau, p_value = scipy.stats.kendalltau(matrix_1, matrix_2)     
        result['tau'] = tau
        result['p_value'] = p_value
        fun = numpy.array([tau,p_value])
        return fun