Home > Software design >  How to append items to a list in a parallel process (python-asyncio)?
How to append items to a list in a parallel process (python-asyncio)?

Time:10-03

I have a function which adds items to the list and returns the list. The items are returned from async function. Now it creates the item and then adds it one by one.

I want to create the items in parallel and add them to the list and after that return the value of the function. How can I solve this?

Thank you in advance!

    async def __create_sockets(self):
        rd_data = []
        for s in self.symbols.index:
            try:
                print(f'Collecting data of {s}')
                socket = DepthCacheManager(self.client, s, refresh_interval=None)
                rd_data.append(await socket.__aenter__())
            except:
                continue
        return rd_data

CodePudding user response:

An easy solution to your problem is to gather the results asynchronously and compile the list of results at the same time.

This is provided by the asyncio.gather() call as explained in the asyncio documentation. Have a look at the excellent example given there.

In your case it might roughly look like this (obviously I cannot test it):

async def create_socket(self, s):
    print(f'Collecting data of {s}')
    socket = DepthCacheManager(self.client, s, refresh_interval=None)
    return socket.__aenter__()

async def __create_sockets(self):
    rd_data = await asyncio.gather(
        *[self.create_socket(s) for s in self.symbols.index]
    )
    return rd_data

There is a problem here with missing exception handling. You may return None in case of an exception and then clean up the list later like this:

async def create_socket(self, s):
    try:
        print(f'Collecting data of {s}')
        socket = DepthCacheManager(self.client, s, refresh_interval=None)
        return await socket.__aenter__() # await is important here
    except:
        return None

async def __create_sockets(self):
    rd_data = await asyncio.gather(
        *[self.create_socket(s) for s in self.symbols.index]
    )
    return [i for i in rd_data if i != None]
  • Related