Python how to use multiprocessing to different functions with return-CodePudding

I have several 3 small functions:

def getnpx(mt, age, interest):
    val = 1
    initval = 1
    for i in range(age, 55):
        val = val * mt[i]
        intval = val / (1   interest) ** (i   1 - age)
        initval = initval   intval
        
    return initval

def getnpx2(mt, age, interest):
    val = mt[age]
    initval = 1
    for i in range(age   2, 55):
        val *= mt[i - 1]
        if mt[age]==0:
            intval =0
        else:
            intval = val / (1   interest) ** (i - age - 1) / mt[age]
        initval = initval   intval
    return initval

def getnpxtocert(mt, age, maxvalue):
    val = mt[age]
    for i in range(age   1, min(maxvalue, 7)):
        val = val * mt[i]
    return val

And 1 large function which call all the small functions:

def calcannfactprelim(pval, age, intrate, certper):
    npx = getnpx(pval, age   int(certper), intrate)
    npx2 = getnpx2(pval, age   int(certper), intrate)

    if certper == 0:
        index = 1
        index2 = pval[age   int(certper)]

    else:
        index = getnpxtocert(pval, age,
                             age   int(certper)) 
        index2 = getnpxtocert(pval, age,age   int(certper)   1)

    return index*npx index2*npx2

These are the variables to use:

pval = np.array([0.000291,0.00027,0.000257,0.000294,0.000325,0.00035,0.000371,0.000388,0.000402,0.000414,0.000425,0.000437,0.011016,0.012251,0.013657,0.015233,0.016979,0.018891,0.020967,0.023209,0.025644,0.028304,0.03122,0.034425,0.037948,0.041812,0.046037,0.050643,0.055651,0.06108,0.066948,0.073275,0.080076,0.08737,0.095169,0.103455,0.112208,0.121402,0.131017,0.14103,0.151422,0.162179,0.173279,0.184706,0.196946,0.210484,0.225806,0.243398,0.263745,0.287334,0.314649,0.346177,0.382403,0.423813,0.470893])
age=3
intrate=0.04
certper=1

Regular function test result:

start=time.time()
print(calcannfactprelim(pval, age, intrate, certper))
print(time.time()-start)

Output is:

0.0002941874880982305 #result
0.0                   #time

In order to make this function faster ,I'm using python multiple processing to parallelly run it.

import multiprocessing

def calcannfactprelim_v(pval, age, intrate, certper):
    
    p1 = multiprocessing.Process(target=getnpx, args=(pval, age, intrate, certper,))
    p2 = multiprocessing.Process(target=getnpx2, args=(pval, age, intrate, certper,))

    # starting process 1
    p1.start()
    # starting process 2
    p2.start()

    # wait until process 1 is finished
    p1.join()
    # wait until process 2 is finished
    p2.join()

    # both processes finished
    if certper == 0:
        index = 1
        index2 = pval[age   int(certper)]

    else:
        index = getnpxtocert(pval, age,
                             age   int(certper)) 
        index2 = getnpxtocert(pval, age,age   int(certper)   1)


    return index*npx index2*npx2

But I have no idea how to return the value in this situation, any friends can help to complete the code and give a test?

CodePudding user response：

You need some channel to return the data. This is done for you in a multiprocessing pool.

def calcannfactprelim_v(pval, age, intrate, certper):
    with multiprocessing.pool(2) as pool:
        a1 = pool.apply_async(getnpx, (pval, age, intrate, certper,))
        a2 = pool.apply_async(getnpx2, (pval, age, intrate, certper,))
        result1 = a1.get()
        result2 = a2.get()

    # both processes finished
    if certper == 0:
        index = 1
        index2 = pval[age   int(certper)]

    else:
        index = getnpxtocert(pval, age,
                             age   int(certper)) 
        index2 = getnpxtocert(pval, age,age   int(certper)   1)
    return index*npx index2*npx2

You could even create a pool of 1 and do one of the calls in the main program. Suppose one returned a large value and the other a small one. Do the large one in your main so that the overhead of getting the value from the process back to the parent is less.

CodePudding user response：

Create an object in the parent process, say,

class ResultHolder():
     def __init__():
          result_a = None
          ………
          result_n = None

Pass this class as an argument to the functions which are to be run in the child processes and let the functions write the result in, say:

instance_of_result_holder.result_a = the_function_output

I believe you can do that by passing a dictionary or a list too. Long story short, objects are passed in a way akin to pass-by-reference, which means you can use them as intermediaries between the two separate worlds (contexts) of the parent and child processes (i.e. a child process can mutate an object defined in the parent one of you pass the latter as an argument to the respective function).

Pools and queues are another way to achieve that.