AsyncIO When does the second await occur in this task?-CodePudding

I have a python script which makes http requests asynchronously, provided a lists of urls, and here is the example code.

#!/usr/local/bin/python3
import httpx
import asyncio

async def httpx_get(client, url):
    resp1 = await client.get(url, timeout=5, allow_redirects=True) 
    # ... other computation ...
    resp2 = await client.get(url, timeout=5, allow_redirects=True) # what happens during this second await?

async def fetch_pages(urls):
    async with httpx.AsyncClient(verify=False) as client:
        # create task for all URLs
        coros = []
        for url in urls:
            coros.append(asyncio.create_task(httpx_get(client, url)))

        # await all tasks
        for coro in asyncio.as_completed(coros):
            resp = await coro 

def main():
    urls = [ "https://google.com", "https://yahoo.com", "https://youtube.com" ] 
    asyncio.run(fetch_pages(urls))

if __name__ == "__main__":
    main()

I realized there is a lack of understanding on my part with when asyncio would handle the second await, resp2 = await client.get(.., in my httpx_get function.

Since for coro in asyncio.as_completed(coros): resp = await coro only resolves the first await, resp1 = await client.get, for each coroutine, when does the second await resp2 = await client.get get resolved?

is the second await/request performed and waited for all at once? or does something else happen? Any clarification would be appreciated, thanks!

CodePudding user response：

...when asyncio would handle the second await, resp2 = await client.get(.., in my httpx_get function.

At the first await the eventloop will pause this task and go on to do other things, eventually checking if client.get has returned. If it has this task will continue executing till it encounters the second await resp2 =await... the eventloop handles this second await like the first - it suspends the task and goes on to do other things till it checks to see if the coroutine has returned.

Here is an example substituting a time delay for retrieving the url - hopefully it is analogous and will show how things might evolve for your process.

import asyncio,random,time

async def httpx_get(client, url):
    print(f'{time.time()} task:{client} resp1 will be called')
    resp1 = await asyncio.sleep(random.randrange(1,10))
    # ... other computation ...
    print(f'{time.time()} task:{client} resp1 returned')
    print(f'{time.time()} task:{client} executing')
    print(f'{time.time()} task:{client} executing')
    print(f'{time.time()} task:{client} executing')
    print(f'{time.time()} task:{client} executing')
    print(f'{time.time()} task:{client} resp2 will be called')
    resp2 = await asyncio.sleep(random.randrange(1,10))
    print(f'{time.time()}  task:{client} resp2 returned')
    return f'{time.time()} task:{client} finished - url'

async def fetch_pages(urls):
    # create task for all URLs
    coros = []
    for n,url in enumerate(urls):
        coros.append(asyncio.create_task(httpx_get(n, url)))

    # await all tasks
    for coro in asyncio.as_completed(coros):
        resp = await coro 
        print(f'{time.time()} {resp}')

def main():
    urls = [ "https://google.com", "https://yahoo.com", "https://youtube.com" ] 
    asyncio.run(fetch_pages(urls))

if __name__ == "__main__":
    main()

Result from an execution:

1659744155.1480112 task:0 resp1 will be called and awaited
1659744155.1480112 task:1 resp1 will be called and awaited
1659744155.1480112 task:2 resp1 will be called and awaited
1659744161.152556 task:2 resp1 returned
1659744161.152556 task:2 executing
1659744161.152556 task:2 executing
1659744161.152556 task:2 executing
1659744161.154198 task:2 executing
1659744161.154198 task:2 resp2 will be called and awaited
1659744163.1660972 task:1 resp1 returned
1659744163.1660972 task:1 executing
1659744163.1660972 task:1 executing
1659744163.1660972 task:1 executing
1659744163.1660972 task:1 executing
1659744163.1675234 task:1 resp2 will be called and awaited
1659744164.1542242 task:0 resp1 returned
1659744164.1542242 task:0 executing
1659744164.1542242 task:0 executing
1659744164.1542242 task:0 executing
1659744164.155838 task:0 executing
1659744164.155838 task:0 resp2 will be called and awaited
1659744166.167943  task:2 resp2 returned
1659744166.167943 1659744166.167943 task:2 finished - https://youtube.com
1659744168.1679864  task:1 resp2 returned
1659744168.1685212 1659744168.1685212 task:1 finished - https://yahoo.com
1659744169.1565957  task:0 resp2 returned
1659744169.1565957 1659744169.1565957 task:0 finished - https://google.com