I just discovered new features of Python 3.11 like ExceptionGroup and TaskGroup and I'm confused with the following TaskGroup behavior: if one or more tasks inside the group fails then all other normal tasks are cancelled and I have no chance to change that behavior Example:

async def f_error():
    raise ValueError()

async def f_normal(arg):
    print('starting', arg)
    await asyncio.sleep(1)
    print('ending', arg)


async with asyncio.TaskGroup() as tg:
    tg.create_task(f_normal(1))
    tg.create_task(f_normal(2))
    tg.create_task(f_error())

# starting 1
# starting 2
#----------
#< traceback of the error here >

In the example above I cannot make "ending 1" and "ending 2" to be printed. Meanwhile it will be very useful to have something like asyncio.gather(return_exceptions=True) option to do not cancel the remaining tasks when an error occurs.

You can say "just do not use TaskGroup if you do not want this cancellation behavior", but the answer is I want to use new exception groups feature and it's strictly bound to TaskGroup

So the questions are:

1. May I somehow utilize exception groups in asyncio without this all-or-nothing cancellation policy in TaskGroup?
2. If for the previous the answer is "NO": why python developers eliminated the possibility to disable cancellation in the TaskGroup API?

CodePudding user response：

BaseExceptionGroups became part of standard Python in version 3.11. They are not bound to asyncio TaskGroup in any way. The documentation is here: https://docs.python.org/3/library/exceptions.html?highlight=exceptiongroup#ExceptionGroup.

Regarding your question 2, within the TaskGroup context you always have the option of creating a task using asyncio.create_task or loop.create_task. Such tasks will not be part of the TaskGroup and will not be cancelled when the TaskGroup closes. An exception in one of these tasks will not cause the group to close, provided the exception does not propagate into the group's __aexit__ method.

You also have the option of handling all errors within a Task. A Task that doesn't propagate an exception won't cancel the TaskGroup.

There's a good reason for enforcing Task cancellation when the group exits: the purpose of a group is to act as a self-contained collection of Tasks. It's contradictory to allow an uncancelled Task to continue after the group exits, potentially allowing tasks to leak out of the context.

CodePudding user response：

As answered by Paul Cornelius, the TaskGroup class is carefully engineered to cancel itself and all its tasks when the first task in it (registered with tg.create_task) raises any Exception.

My understanding that a "forgiveful" task group, that would await for all other tasks upon it's context exit (end of async with block), regardless of ne or more tasks created in it erroring would still be useful, and that is the functionality you want.

I tinkered around the source code for the TaskGroup, and I think the minimal coding to get the forgiveful task group can be achieved by neutering its internal _abort method. Keep in mind that as _abort starts with an underscore, it is an implementation detail, and the mechanisms for aborting might change inside TaskGroup even during Py 3.11 lifetime.

For now, I could get it working like this:

import asyncio

class ForgivingTaskGroup(asyncio.TaskGroup):
    _abort = lambda self: None

async def f_error():
    print("starting error")
    raise RuntimeError("booom")

async def f_normal(arg):
    print('starting', arg)
    await asyncio.sleep(.1)
    print('ending', arg)

async def main():
    async with ForgivingTaskGroup() as tg:
        tg.create_task(f_normal(1))
        tg.create_task(f_normal(2))
        tg.create_task(f_error())

        # await asyncio.sleep(0)

asyncio.run(main())

The stdout I got here is:

starting 1
starting 2
starting error
ending 1
ending 2

And stderr displayed the beautiful ASCII-art tree as by the book, but with a single exception as child.