When Do You Need to Compile CPython Yourself?

Most operating systems provide compiled CPython versions. Generally, installing through a package manager is sufficient. However, in some cases, you need to compile CPython yourself to meet specific requirements:

  • The Python version provided by the OS is too old, and the Python official website or system package repositories do not provide pre-compiled new versions of Python.

  • Pre-compiled versions do not meet performance or extension requirements, for example, compiler optimizations are not enabled, or OpenSSL/SQLite versions do not meet requirements.

  • Participating in CPython development or trying out new features, such as Alpha/Beta/RC versions of Python.

Precautions When Compiling CPython on Older Linux Distributions

OpenSSL

Because the OpenSSL version in the official CentOS 6 repository is too old and does not meet the requirements of Python 3.7 and later, running simple configure & make will result in an error. Solution:

Refer to Python 3.7 on CentOS 6, compile OpenSSL in advance, modify the Modules/Setup file when compiling CPython, and specify the environment variable LDFLAGS="-Wl,-rpath=/usr/local/openssl11/lib" and argument -with-openssl=/usr/local/openssl11.

SQLite

Software like Jupyter relies on SQLite. When compiling CPython, you should not only pay attention to the SQLite version but also enable --enable-loadable-sqlite-extensions. Refer to: How to use enable_load_extension from sqlite3?

Compilation Parameters Affecting Performance

–enable-optimizations

According to this StackOverflow question: what does –enable-optimizations do while compiling python?

enable-optimizations enables PGO (Profile Guided Optimization) and LTO (Link Time Optimization) compiler optimizations. PGO performs two compilations. After the first compilation, it runs benchmark tests, and guides the second optimization compilation based on the tests.

The advantage is that it produces a more efficient CPython binary.

The disadvantage is that it firstly requires a certain compiler version (older GCC versions may error out during PGO compilation), and secondly, the compilation time increases significantly due to the benchmarks and the second compilation. Therefore, consider adding the -j parameter when running make to utilize multi-cores and reduce compilation time.

-march

Specifying the CPU architecture may also affect the performance of the compiled CPython. Extended reading: Compile-time Optimisations on Python 3

-fno-semantic-interposition

Refer to Red Hat Enterprise Linux 8.2 brings faster Python 3.8 run speeds.

Adding the -fno-semantic-interposition option during GCC compilation disables Semantic interposition on Red Hat / Fedora distributions, bringing up to a 30% performance improvement.

Of course, the above compilation parameters are related to specific distributions and compiler versions. You cannot simply assume they apply to all environments. The best way to evaluate performance is still to benchmark.