How to convert a python for loop from sequential to run parallel-CodePudding

I just learnt about asynchronous and parallel functions in python. Where I can run a for loop in parallel and the functions can run in parallel instead of waiting for the first one to end before calling the next one.

I have the the following function which I want to run in parallel over the alpaca trading platform.

I have multiple accounts and I want some trades to execute in parallel over them.

The code below is just a simple code to get information about the accounts. My final code will be more complex.

How can I run it in parallel?

import alpaca_trade_api as tradeapi

keys = [['apikey1','secretkey1'],['apikey2','secretkey2'],'apikey3','secretkey3']

for key in keys:
    print(key)
    api = tradeapi.REST(key[0],key[1])
    conn = tradeapi.StreamConn(key[0],key[1])
    account = api.get_account()
    account.status

    print(float(account.equity))
    print(float(account.last_equity))
    print(float(account.equity) - float(account.last_equity))

So to run it in parallel I tried this,

import multiprocessing

def printstats(key):
    print(key)
    api = tradeapi.REST(key[0],key[1])
    conn = tradeapi.StreamConn(key[0],key[1])
    account = api.get_account()
    account.status

    print(float(account.equity))
    print(float(account.last_equity))
    print(float(account.equity) - float(account.last_equity))


a_pool = multiprocessing.Pool()

result = a_pool.map(printstats, keys)

print(result)

But it just runs forever and doesn't print anything.

If I run that for loop normally it works.

What am I doing wrong? Should I be using async instead of this? What's the difference?

CodePudding user response：

For the multiprocessing module you can't start the sub-processes on the top-level of code. If you just wrap those 3 lines to make them only run when the script is executed vs whenever it is called / imported the sub-processes will be able to spawn.

if __name__ == '__main__':
    a_pool = multiprocessing.Pool()
    result = a_pool.map(printstats, keys)
    print(result)

Newer syntax that will automatically close the process pool once it leaves the code block.

if __name__ == '__main__':
    with multiprocessing.Pool() as pool:
        res = pool.map(printstats, keys)
        print(res)

Note about async / await:

Async / await is for asynchronous code (non-blocking) not parallel code.

If you'd like to read more this article covers all the different terminology: https://phuctm97.com/blog/sync-async-concurrent-parallel