This article explains the important new features of Python 3.11, but there are still many small changes, so I won’t mention them one by one, so if you’re interested, you should read the official changelog to understand them.
This is the most exciting news ever. The official website says :
CPython 3.11 is on average 25% faster than CPython 3.10 when measured with the pyperformance benchmark suite, and compiled with GCC on Ubuntu Linux. Depending on your workload, the speedup could be up to 10-60% faster.
That is, the speedup is between 10-60%, or 25% faster than Python 3.10 on average, which is a pretty big improvement.
There’s actually quite a story to this sudden improvement in Python.
It started in late 2020 when Mark Shannon, a little-known Python developer, came up with a project called “A faster CPython”, in which he planned to make CPython 5x faster in the next 4 new versions of Python. In his plan, he listed some of the optimizations he had found in his Python implementation, and also listed his plans for each new Python version. This plan was a big hit in the community for a few days.
Then Guido van Rossum, the father of Python, announced at the Python Language Summit in May 2021 that he had committed to the project, and that he had assembled a small team funded by Microsoft, along with Eric Snow, a well-known Python core developer, and Mark Shannon, the project’s proponent. The very good news is that Mark Shannon has now officially joined Microsoft, and I have to thank Microsoft for their support of open source and Python. 👍🏻
So this time the speed improvement mainly comes from this plan. Some of the most significant work is as follows.
- PEP 659 - Specializing Adaptive Interpreter. since the types of objects rarely change, the interpreter now tries to analyze the running code and replace the generic bytecode with type-specific bytecode. For example, binary operations (addition, subtraction, etc.) can be replaced with specialized versions of integers, floating point numbers, and strings.
- Function calls have less overhead. The stack frames for function calls now use less memory and are designed to be more efficient.
- Core modules needed at runtime can be stored and loaded more efficiently.
- ‘Zero overhead’ exception handling.
As you can see, version 3.11 is a good start to speeding up Python. We’ll talk more about the goals and plans for Python 3.12 in due course.
PEP 654 - Exception Groups and except*
A new syntax has been introduced in PEP 654 to make exception handling simpler. Normally only one error occurs at a time, the code runs due to program logic or external factors throwing errors, and then prints a stack of error messages, which is intuitive. This PEP also lists 5 types, among which the
Hypothesis library referenced in
Multiple errors in complex computational processes is a library I have no experience in using and can’t find an actual example, but the other types are given as corresponding examples.
1. Multiple concurrent tasks may fail simultaneously
For example, programs written by the asyncio library.
I am demonstrating here, suppose now I have written a web crawler using asyncio and aiohttp, only
core_success succeeds for the 4 tasks, the others will throw different errors. Note that you must not use
return_exceptions=False which will lose the exception information.
The problem with this usage is that you can’t catch exceptions directly when
gather executes the task, you need to get the exceptions from this list only after the end of execution and traversing the results.
2. The cleanup code also happens with its own errors
Multiple user callbacks fail and
Errors in wrapper code mentioned in the PEP into this article. For example,
atexit.register(), with the
__exit__ at the end of the code block.
We were trying to catch a developer-defined exception, but the exception was not caught because of an error thrown on the context cleanup exit.
3. Different kinds of errors are hidden in error retries
Here is an abbreviated version of the standard library
We know that network requests can have various types of errors, and here only the last error message is kept, and the previous ones are ignored.
Introduction of syntax
The improvement solution is to introduce exception groups, which are a structure that can carry subexceptions.
Let’s look at one more complex structure.
Above I built a total of three levels of exception groups, you feel the structure and flexibility of this syntax, through the different levels can show the complex exception information.
Then look at the other syntax
except*. It is used to match
ExceptionGroup, and the asterisk symbol (
*) indicates that each
except* clause can handle multiple exceptions:
Let’s look at a more obvious example.
TypeError at both the first and second level, and if you use the old
except it will fail to catch.
Another major feature of this new syntax is that it can catch a series of exceptions, so if exception A is a subclass of B of some exception, then you can also get A by catching B. It’s a little hard to understand, so let’s give an example.
Among the above 4 exceptions, all of them are subclasses of
IndexError, so they can be caught together by
except* OSError. Also note that catching specific types later will fail .
except* ConnectionError will not succeed because it will meet the previous conditions to not get here.
PEP 678 - Enriching Exceptions with Notes
PEP 678 proposes to add an
add_note method to an exception to add notes to it, which will be recorded in the
You can also use
add_notes inside the exception group :
That is, the added comment will appear below the exception message.
PEP 657: Fine-grained error locations in tracebacks
I think this is similar to the work done in Python 3.10 in the direction of “better error hints”, and there’s a little bit of a story about this improvement.
On 7/22/19, Guido van Rossum, the father of Python, wrote a blog post on Medium called “PEG Parsers”. It says that he is considering refactoring the Python interpreter using PEG Parser instead of the existing class LL (1) Parser. The reason is that the current pgen limits the freedom of Python syntax, making some syntax difficult to implement and making the current syntax tree less tidy, to some extent affecting the ideation of the syntax tree and not best reflecting the designer’s intent.
He wrote several articles about this PEG in quick succession, and Python 3.9 already had a new parser based on PEG (Parsing Expression Grammar) instead of LL (1). The performance of the new parser is roughly equivalent to the old one, but PEG is more flexible than LL (1) in designing new language features for the formalism.
It is thanks to this PEG that the match-case syntax was added to Python and a lot of improvements were made to better error messages.
In the past, when an exception occurs, the input data structure is simple to quickly find the point of the problem, but when dealing with complex structures for the location of the code where the error occurred is reported inaccurately. This definitely makes it more difficult for junior developers to solve problems, and several specific examples are listed on the official website. As with Python 3.10, it’s not necessary to understand the improvements to each error tip, but it’s good to know that Python 3.11 is more accurate at locating error points in Traceback.
For more details, see my previous article: New Type System-related Features in Python 3.11
AsyncIO Task Groups
asyncio.TaskGroup will be used in the future to replace the previous
asyncio.gather API, which has the main advantage of supporting the new exception group feature to catch asynchronous IO exceptions. Let me give you an example to compare.
gather will lose exceptions if the argument is
return_exceptions=False. Take a look at the example of
These two examples implement the same logic, but
asyncio.TaskGroup is more flexible, in the context of
async with, new tasks can be added at any time as long as all tasks are not completed. Another advantage of task groups is that they are more flexible to cancel tasks.
In the above example, a new
core_long task sleeps for 1 second, and does not wait for the end of execution to determine the task status directly, so it can be easily cancelled if it does not complete.