Creating an async iterator for any iterable-CodePudding

I am creating a project which makes many calls to external APIs. These API calls are made within methods of class instances. I am attempting to make a generic function which takes an iterable of these objects and produces an async iterator for them. This async iterator will then be used to make all of these external API calls run asynchronously.

However, with what I have tried below, the exectution time still increases linearly with the length of the list.

async def async_iterator(iterable: Iterable[Any]) -> AsyncIterator[Any]:
    for i in iterable:
        yield i

async for object in async_generator(iterable=list_of_objects):
    await object.make_time_consuming_api_call()
    # do other work on the object
    await update_in_database(object=object)

How can I asynchronously iterate over any list of objects?

CodePudding user response：

Since you're awaiting the object.make_time_consuming_api_call(), it waits for each call to be completed before the next iteration can be run. You could just await it later after submitting all the calls with something like asyncio.create_task:

async def async_iterator(iterable: Iterable[Any]) -> AsyncIterator[Any]:
    for i in iterable:
        yield i

async def main():
    tasks = list()
    async for object in async_generator(iterable=list_of_objects):
        tasks.append(asyncio.create_task(object.make_time_consuming_api_call()))

    # do other work on the object
    for task, object in zip(tasks, list_of_objects):
        await task
        await update_in_database(object=object)

In this case, you won't even need to create an async_iterator:

async def main():
    tasks = list()
    for object in list_of_objects:
        tasks.append(asyncio.create_task(object.make_time_consuming_api_call()))

Or to make it even more concise:

async def main():
    results = await asyncio.gather(*(object.make_time_consuming_api_call() for object in list_of_objects))

    # Added this to store the result as an attribute (see comments)
    for result, object in zip(results, list_of_objects):
        object.attribute = result