I recently wanted to check out the new pattern matching features in Python 3.10, and I just happened to understand some of the basics of compiling, so I tried to compile a copy of the latest Python on the test machine I usually use.

Working environment

The operating system of the test machine is Ubuntu 20.10, with basic development tools such as gcc and git already installed. The following steps have been verified to work with macOS 11 as well.

1
2

uname -a
Linux waynerv-woqutech 5.8.0-48-generic #54-Ubuntu SMP Fri Mar 19 14:25:20 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Download source code

Clone the source code from CPython’s official repository at https://github.com/python/cpython.

Configuration

The core CPython compiler only requires a basic C compiler to compile, but some extension modules will require development headers to provide some additional libraries (such as the zlib library needed for the compression feature). That’s why we need python3-dev and a bunch of other dependencies when we install Python with an OS-provided package manager like apt.

Since I don’t intend to use my self-compiled Python for development or production, I’ll skip this step of installing the dependencies.

First, go to the directory of the repository we cloned.

1

cd cpython

Then it needs to be configured.

1

./configure --prefix=/root/build-python

By default, the subsequent make install command will install the compiled files to /usr/local/bin or /usr/local/lib, which may overwrite the system’s existing installation files, so as not to conflict with the system’s installed version of Python, we install it to a custom directory by specifying the -prefix option.

Compile

Run the compilation after the configuration.

1

make -s -j 4

You can speed up the compilation by specifying the number of parallel tasks with the -j option, which is usually set to the number of CPUs on the compiling machine and can be used in conjunction with the nproc command.

1

make -s -j "$(nproc)"

The -s option, meaning silence, which does not print the compiled process log, can also be enabled.

The compilation process can be time consuming and the output upon success is as follows.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

Python build finished successfully!
The necessary bits to build these optional modules were not found:
_bz2                  _curses               _curses_panel      
_dbm                  _gdbm                 _hashlib           
_lzma                 _sqlite3              _ssl               
_tkinter              _uuid                 readline           
To find the necessary bits, look in setup.py in detect_modules() for the module's name.


The following modules found by detect_modules() in setup.py, have been
built by the Makefile instead, as configured by the Setup files:
_abc                  pwd                   time               


Could not build the ssl module!
Python requires an OpenSSL 1.0.2 or 1.1 compatible libssl with X509_VERIFY_PARAM_set1_host().
LibreSSL 2.6.4 and earlier do not provide the necessary APIs, https://github.com/libressl-portable/portable/issues/381

Since we don’t have dependencies installed, we are prompted with some modules not found, but this does not affect the basic functionality of using Python.

Installation

You can actually run Python after a successful compilation. The above build process generates a binary file called python (python.exe on macOS) in the current directory (not the directory specified by -prefix), which you run to start the Python interpreter.

1

./python

It will use the temporary files generated by the compilation in the current directory as resource files, which we will now install into the directory specified in the configuration step.

1

make install

The installation process does a lot of resource copying and calls the Python interpreter to compile the standard libraries implemented in Python to bytecode ( .pyc), in addition to installing the setuptools and pip Python packages by default, so that when we install newer versions of Python, we basically no longer need to manually install pip .

Once the installation is complete, switch to the /root/build-python directory and look at its directory structure.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

cd /root/build-python
tree -L 2 .
.
├── bin
│   ├── 2to3 -> 2to3-3.10
│   ├── 2to3-3.10
│   ├── idle3 -> idle3.10
│   ├── idle3.10
│   ├── pip3
│   ├── pip3.10
│   ├── pydoc3 -> pydoc3.10
│   ├── pydoc3.10
│   ├── python3 -> python3.10
│   ├── python3.10  # Python 可执行文件
│   ├── python3.10-config
│   └── python3-config -> python3.10-config
├── include
│   └── python3.10  # 头文件目录
├── lib
│   ├── libpython3.10.a  # 静态库文件
│   ├── pkgconfig
│   └── python3.10  # 标准库文件及 site-packages
└── share
    └── man

Now run . /bin/python3 to run our own compiled and installed Python.

1
2
3

Python 3.10.0a6+ (heads/master:80017752ba, Apr  6 2021, 13:47:23) [GCC 10.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

Done! You can see that you are running Python version 3.10.0a6+, a version that is not yet released for development.

If we need another stable version of Python, we can simply git checkout to the specified release tag in the source repository, and then re-run the configure-compile-install 3-step process.

Reference https://www.waynerv.com/posts/compiling-a-latest-python/