Useful new features or functionality in Python 3.11

Preface

This article explains the important new features of Python 3.11, but there are still many small changes, so I won’t mention them one by one, so if you’re interested, you should read the official changelog to understand them.

Speed improvements

This is the most exciting news ever. The official website says :

CPython 3.11 is on average 25% faster than CPython 3.10 when measured with the pyperformance benchmark suite, and compiled with GCC on Ubuntu Linux. Depending on your workload, the speedup could be up to 10-60% faster.

That is, the speedup is between 10-60%, or 25% faster than Python 3.10 on average, which is a pretty big improvement.

There’s actually quite a story to this sudden improvement in Python.

It started in late 2020 when Mark Shannon, a little-known Python developer, came up with a project called “A faster CPython”, in which he planned to make CPython 5x faster in the next 4 new versions of Python. In his plan, he listed some of the optimizations he had found in his Python implementation, and also listed his plans for each new Python version. This plan was a big hit in the community for a few days.

Then Guido van Rossum, the father of Python, announced at the Python Language Summit in May 2021 that he had committed to the project, and that he had assembled a small team funded by Microsoft, along with Eric Snow, a well-known Python core developer, and Mark Shannon, the project’s proponent. The very good news is that Mark Shannon has now officially joined Microsoft, and I have to thank Microsoft for their support of open source and Python. 👍🏻

So this time the speed improvement mainly comes from this plan. Some of the most significant work is as follows.

PEP 659 - Specializing Adaptive Interpreter. since the types of objects rarely change, the interpreter now tries to analyze the running code and replace the generic bytecode with type-specific bytecode. For example, binary operations (addition, subtraction, etc.) can be replaced with specialized versions of integers, floating point numbers, and strings.
Function calls have less overhead. The stack frames for function calls now use less memory and are designed to be more efficient.
Core modules needed at runtime can be stored and loaded more efficiently.
‘Zero overhead’ exception handling.

As you can see, version 3.11 is a good start to speeding up Python. We’ll talk more about the goals and plans for Python 3.12 in due course.

PEP 654 - Exception Groups and except*

A new syntax has been introduced in PEP 654 to make exception handling simpler. Normally only one error occurs at a time, the code runs due to program logic or external factors throwing errors, and then prints a stack of error messages, which is intuitive. This PEP also lists 5 types, among which the Hypothesis library referenced in Multiple errors in complex computational processes is a library I have no experience in using and can’t find an actual example, but the other types are given as corresponding examples.

1. Multiple concurrent tasks may fail simultaneously

For example, programs written by the asyncio library.

import aiohttp
import asyncio

async def core_success():
    return('success')


async def core_error1():
    raise asyncio.TimeoutError


async def core_error2():
    raise aiohttp.ClientOSError


async def core_error3():
    raise aiohttp.ClientConnectionError


async def raise_errors():
    results = await asyncio.gather(
        core_success(),
        core_error1(),
        core_error2(),
        core_error3(),
        return_exceptions=True)
    for r in results:
        match r:
            case asyncio.TimeoutError():
                print('timeout_error')
            case aiohttp.ClientOSError():
                print('client_os_error')
            case aiohttp.ClientConnectionError():
                print('client_connection_error')
            case _:
                print(r)


asyncio.run(raise_errors())

I am demonstrating here, suppose now I have written a web crawler using asyncio and aiohttp, only core_success succeeds for the 4 tasks, the others will throw different errors. Note that you must not use return_exceptions=False which will lose the exception information.

The problem with this usage is that you can’t catch exceptions directly when gather executes the task, you need to get the exceptions from this list only after the end of execution and traversing the results.

2. The cleanup code also happens with its own errors

I put Multiple user callbacks fail and Errors in wrapper code mentioned in the PEP into this article. For example, atexit.register(), with the __exit__ at the end of the code block.

from contextlib import ContextDecorator

class mycontext(ContextDecorator):
    def __enter__(self):
        return self

    def __exit__(self, *exc):
        raise RuntimeError('hah')
        return False


try:
    with mycontext() as c:
        raise TypeError()
except TypeError:
    print('catch!')

We were trying to catch a developer-defined exception, but the exception was not caught because of an error thrown on the context cleanup exit.

Traceback (most recent call last):
  File "/Users/weiming.dong/mp/2022-10-24/cleanup.py", line 14, in <module>
    raise TypeError()
TypeError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/weiming.dong/mp/2022-10-24/cleanup.py", line 13, in <module>
    with mycontext() as c:
  File "/Users/weiming.dong/mp/2022-10-24/cleanup.py", line 8, in __exit__
    raise RuntimeError('hah')
RuntimeError: hah

3. Different kinds of errors are hidden in error retries

Here is an abbreviated version of the standard library socket.create_connection implementation.

def create_connection(address, timeout=_GLOBAL_DEFAULT_TIMEOUT,
                      source_address=None):
    host, port = address
    err = None
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
        try:
            # do some stuff
            return sock

        except error as _:
            err = _
            if sock is not None:
                sock.close()

    if err is not None:
        try:
            raise err
        finally:
            err = None
    else:
        raise error("getaddrinfo returns an empty list")

We know that network requests can have various types of errors, and here only the last error message is kept, and the previous ones are ignored.

Introduction of syntax

The improvement solution is to introduce exception groups, which are a structure that can carry subexceptions.

In : eg = ExceptionGroup('request fail', [OSError('bad path'), ImportError('not found')])

In : eg.exceptions  # The exception tuple, by which you can find the corresponding exception, is a very important property
Out: (OSError('bad path'), ImportError('not found'))

In : eg.message
Out: 'request fail'

In : eg.args
Out: ('request fail', [OSError('bad path'), ImportError('not found')])

In : eg.split(OSError)  # Split groups according to error type
Out:
(ExceptionGroup('request fail', [OSError('bad path')]),
 ExceptionGroup('request fail', [ImportError('not found')]))

Let’s look at one more complex structure.

In : exc = ExceptionGroup('d2 exc', [RuntimeError(1)])

In : exc2 = ExceptionGroup('d1 exc', [TypeError(2), exc])

In : exc3
Out:
ExceptionGroup('root exc',
               [ExceptionGroup('d1 exc',
                               [TypeError(2),
                                ExceptionGroup('d2 exc', [RuntimeError(1)])]),
                OSError(3),
                IndexError(4)])

In : exc3.exceptions[0].exceptions[1].exceptions[0]
Out: RuntimeError(1)

In : err = exc3.subgroup(lambda e: isinstance(e, RuntimeError))  # Conditional Filtering

In : err
Out:
ExceptionGroup('root exc',
               [ExceptionGroup('d1 exc',
                               [ExceptionGroup('d2 exc', [RuntimeError(1)])])])

In : import traceback

In : traceback.print_exception(err)
  | ExceptionGroup: root exc (1 sub-exception)
  +-+---------------- 1 ----------------
    | ExceptionGroup: d1 exc (1 sub-exception)
    +-+---------------- 1 ----------------
      | ExceptionGroup: d2 exc (1 sub-exception)
      +-+---------------- 1 ----------------
        | RuntimeError: 1
        +------------------------------------

Above I built a total of three levels of exception groups, you feel the structure and flexibility of this syntax, through the different levels can show the complex exception information.

Then look at the other syntax except*. It is used to match ExceptionGroup, and the asterisk symbol (*) indicates that each except* clause can handle multiple exceptions:

try:
    ...
except* SpamError:
    ...
except* FooError as e:
    ...
except* (BarError, BazError) as e:
    ...

Let’s look at a more obvious example.

In : eg
Out:
ExceptionGroup('d1 exc',
               [TypeError(2),
                ExceptionGroup('d2 exc', [TypeError(1), IndexError(3)]),
                RuntimeError(4)])

In : try:
...:     raise eg
...: except TypeError as e:  # Not captured
...:     print(f'catch errors: {e=}')
...: except Exception as e2:
...:     print(f'other errors: {e2=}')
...:
other errors: e2=ExceptionGroup('d1 exc', [TypeError(2), ExceptionGroup('d2 exc', [TypeError(1), IndexError(3)]), RuntimeError(4)])

In : try:
...:     raise eg
...: except* TypeError as e:  # Successfully captured
...:     print(f'catch errors: {e=}')
...: except* Exception as e2:
...:     print(f'other errors: {e2=}')
...:
catch errors: e=ExceptionGroup('d1 exc', [TypeError(2), ExceptionGroup('d2 exc', [TypeError(1)])])
other errors: e2=ExceptionGroup('d1 exc', [ExceptionGroup('d2 exc', [IndexError(3)]), RuntimeError(4)])

There is TypeError at both the first and second level, and if you use the old except it will fail to catch.

Another major feature of this new syntax is that it can catch a series of exceptions, so if exception A is a subclass of B of some exception, then you can also get A by catching B. It’s a little hard to understand, so let’s give an example.

In : eg = ExceptionGroup('root exc',
...:                [ExceptionGroup('d1 exc',
...:                                [BlockingIOError(2),
...:                                 ExceptionGroup('d2 exc', [ConnectionError(1)])]),
...:                 FileExistsError(3),
...:                 IndexError(4)])

In : try:
...:     raise eg
...: except* OSError as e:
...:     print(f'catch errors: {e=}')
...: except* Exception as e2:
...:     print(f'other errors: {e2=}')
...:
catch errors: e=ExceptionGroup('root exc', [ExceptionGroup('d1 exc', [BlockingIOError(2), ExceptionGroup('d2 exc', [ConnectionError(1)])]), FileExistsError(3)])
other errors: e2=ExceptionGroup('root exc', [IndexError(4)])

Among the above 4 exceptions, all of them are subclasses of OSError except IndexError, so they can be caught together by except* OSError. Also note that catching specific types later will fail .

In : try:
...:     raise eg
...: except* OSError as e:
...:     print(f'catch errors: {e=}')
...: except* ConnectionError:  # Newly added
...:     print('not catch!')
...: except* Exception as e2:
...:     print(f'other errors: {e2=}')
...:

Where except* ConnectionError will not succeed because it will meet the previous conditions to not get here.

PEP 678 - Enriching Exceptions with Notes

PEP 678 proposes to add an add_note method to an exception to add notes to it, which will be recorded in the __notes__ attribute.

➜ python  # The latest version of IPython does not yet support this feature, so use the Python interpreter first
>>> try:
...     raise TypeError('Bad type')
... except Exception as e:
...     e.add_note('Add some information')
...     raise
...
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
TypeError: Bad type
Add some information
>>> e.__notes__  # The default exception does not yet have this property
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'TypeError' object has no attribute '__notes__'. Did you mean: '__ne__'?
>>> e.add_note('Add some information')
>>> e.__notes__
['Add some information']

You can also use add_notes inside the exception group :

def task(num):
    raise TypeError(f'Error with {num}')


errors = []
for num in [1, 4, 9]:
    try:
        task(num)
    except TypeError as e:
        e.add_note(f'Note: {num}')
        errors.append(e)

if errors:
    raise ExceptionGroup('Task issues', errors)

Effect:

+ Exception Group Traceback (most recent call last):
  |   File "/Users/weiming.dong/mp/2022-10-24/pep678.py", line 14, in <module>
  |     raise ExceptionGroup("Task issues", errors)
  | ExceptionGroup: Task issues (3 sub-exceptions)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/Users/weiming.dong/mp/2022-10-24/pep678.py", line 8, in <module>
    |     task(num)
    |   File "/Users/weiming.dong/mp/2022-10-24/pep678.py", line 2, in task
    |     raise TypeError(f'Error with {num}')
    | TypeError: Error with 1
    | Note: 1
    +---------------- 2 ----------------
    | Traceback (most recent call last):
    |   File "/Users/weiming.dong/mp/2022-10-24/pep678.py", line 8, in <module>
    |     task(num)
    |   File "/Users/weiming.dong/mp/2022-10-24/pep678.py", line 2, in task
    |     raise TypeError(f'Error with {num}')
    | TypeError: Error with 4
    | Note: 4
    +---------------- 3 ----------------
    | Traceback (most recent call last):
    |   File "/Users/weiming.dong/mp/2022-10-24/pep678.py", line 8, in <module>
    |     task(num)
    |   File "/Users/weiming.dong/mp/2022-10-24/pep678.py", line 2, in task
    |     raise TypeError(f'Error with {num}')
    | TypeError: Error with 9
    | Note: 9
    +------------------------------------

That is, the added comment will appear below the exception message.

PEP 657: Fine-grained error locations in tracebacks

I think this is similar to the work done in Python 3.10 in the direction of “better error hints”, and there’s a little bit of a story about this improvement.

On 7/22/19, Guido van Rossum, the father of Python, wrote a blog post on Medium called “PEG Parsers”. It says that he is considering refactoring the Python interpreter using PEG Parser instead of the existing class LL (1) Parser. The reason is that the current pgen limits the freedom of Python syntax, making some syntax difficult to implement and making the current syntax tree less tidy, to some extent affecting the ideation of the syntax tree and not best reflecting the designer’s intent.

He wrote several articles about this PEG in quick succession, and Python 3.9 already had a new parser based on PEG (Parsing Expression Grammar) instead of LL (1). The performance of the new parser is roughly equivalent to the old one, but PEG is more flexible than LL (1) in designing new language features for the formalism.

It is thanks to this PEG that the match-case syntax was added to Python and a lot of improvements were made to better error messages.

In the past, when an exception occurs, the input data structure is simple to quickly find the point of the problem, but when dealing with complex structures for the location of the code where the error occurred is reported inaccurately. This definitely makes it more difficult for junior developers to solve problems, and several specific examples are listed on the official website. As with Python 3.10, it’s not necessary to understand the improvements to each error tip, but it’s good to know that Python 3.11 is more accurate at locating error points in Traceback.

Type system

For more details, see my previous article: New Type System-related Features in Python 3.11

AsyncIO Task Groups

This asyncio.TaskGroup will be used in the future to replace the previous asyncio.gather API, which has the main advantage of supporting the new exception group feature to catch asynchronous IO exceptions. Let me give you an example to compare.

async def core_success():
    return 'success'


async def core_value_error():
    raise ValueError


async def core_type_error():
    raise TypeError


async def gather():
    results = await asyncio.gather(
        core_success(),
        core_value_error(),
        core_type_error(),
        return_exceptions=True)
    for r in results:
        match r:
            case ValueError():
                print('value_error')
            case TypeError():
                print('type_error')
            case _:
                print(r)

Note that gather will lose exceptions if the argument is return_exceptions=False. Take a look at the example of asyncio.TaskGroup.

async def task_group1():
    try:
        async with asyncio.TaskGroup() as g:
            task1 = g.create_task(core_success())
            task2 = g.create_task(core_value_error())
            task3 = g.create_task(core_type_error())
        results = [task1.result(), task2.result(), task3.result()]
    except* ValueError as e:
        raise
    except* TypeError as e:
        raise


asyncio.run(task_group1())

These two examples implement the same logic, but asyncio.TaskGroup is more flexible, in the context of async with, new tasks can be added at any time as long as all tasks are not completed. Another advantage of task groups is that they are more flexible to cancel tasks.

async def core_success():
    print('success')


async def core_value_error():
    raise ValueError


async def core_long():
    try:
        await asyncio.sleep(1)
        print('long task done!')
    except asyncio.CancelledError:
        print('cancelled!')
        raise

async def task_group2():
    try:
        async with asyncio.TaskGroup() as g:
            task1 = g.create_task(core_success())
            task2 = g.create_task(core_value_error())
            task3 = g.create_task(core_long())
        results = [task1.result(), task2.result(), task3.result()]
    except* ValueError as e:
        print(f'{e=}')
    except* TypeError as e:
        print(f'{e=}')

    for r in [task1, task2, task3]:
        if not r.done():
            r.cancel()

In the above example, a new core_long task sleeps for 1 second, and does not wait for the end of execution to determine the task status directly, so it can be easily cancelled if it does not complete.

➜ python asyncio_task_group.py
success
cancelled!
e=ExceptionGroup('unhandled errors in a TaskGroup', [ValueError()])