How to Build Python from Source

Submitted by nanodano on Thu, 08/01/2019 - 02:21

Introduction

Installing Python is easy using the pre-built installers and packages from your operating system. However, if you want to build the cutting-edge version directly from GitHub master branch, you will have to build your own version from source. You may also want to do it just to reinforce your understanding of Python.

This guide will walk through the steps needed to build Python 3 from source and then create a virtual environment that you can use for projects.

Build steps

This assumes a Linux development environment and was tested in Fedora 30. You may need to install some of the necessary dev tools like make or gcc on your system.

The main steps are:

Obtain the source code
Run the configure script
Run make
Run make install

Get the source code

You can get the latest code by cloning directly from master GitHub branch. At the time of this writing, the Git repository contains Python 3.9.

git clone https://github.com/python/cpython

You can also download the source for official releases on the releases page.

Configure

A configure script comes in the source that can be passed many values. This will create the Makefile. One in particular that is important is the --prefix. This will determine where the final built files go.

cd cpython
./configure --prefix=$HOME/python3.9

By default, it will generate (among other things) a libpython3.9.a static library that you can use to link with C applications (e.g. gcc my.c -lpython3.9). If you want to build the shared library instead of the static one (libpython3.so), the use the --enabled-shared flag. This will NOT build the static library and you will have to ensure the .so file is loadable when running Python which adds more complexity. I recommend avoiding the shared library unless you have a need for it.

./configure --enable-shared

If you want to add optimizations, also add on the --enable-optimizations flag.

./configure --enable-optimizations

An example with all the options might look like this:

./configure --enable-optimizations --prefix=$HOME/python3.9

Build

Use the make tool to build the files followed by make install to put the final files in the location specified by configure's --prefix.

make  # Do the bulk of compilation

Near the end of the output, you should see a success message and some suggestions on optional modules. In the output below it mentions that it could not build _tkinter.

Python build finished successfully!
The necessary bits to build these optional modules were not found:
_dbm                  _gdbm                 _tkinter           
nis                   readline
To find the necessary bits, look in setup.py in detect_modules() for the module's name.


The following modules found by detect_modules() in setup.py, have been
built by the Makefile instead, as configured by the Setup files:
_abc                  atexit                pwd                
time

In Fedora, the optional missing dependencies can be installed with:

dnf install tk-devel readline-devel gdbm-devel

In Debian, the packages are:

apt install libncurses-dev libgdbm-dev libz-dev tk-dev libsqlite3-dev libreadline-dev liblzma-dev libffi-dev libssl-dev

Run ./configure and make again to rebuild with the dependencies.

After the build was successful, compile all the final files in to the target destination that was specified by --prefix on the configure script.

make install  # Puts the final files in prefix location

Post-build

After it performs the install, the prefix directory specified in configure will contain the output, which should be four directories:

├── bin
├── include
├── lib
└── share

The bin dir contains the all-important python3 executable.
The include directory contains all the include files needed for Python dev, including the important Python.h used for writing C extensions. This is the directory you would add to gcc with the -I flag if compiling C applications with embedded Python.
The lib directory has all the Python modules as well as the libpython3.9.a library. Add this library directory to gcc search path for libraries with -L and link to the lib with -lpython3.9.
The share directory will contain the man pages. Can be read with man (e.g. man ./share/man/man1/python3.9.1).

You could run Python directly from the bin/python3 executable, or add bin/ to your PATH environment variable.

Since Python 3 comes with the virtual environment package, I suggest creating a new virtual environment from the freshly built Python.

# Create a virtual environment from the new python
$HOME/mycpython/bin/python3 -m venv $HOME/venv

Then you can activate the virtual environment and ensure everything looks good.

source $HOME/venv/bin/activate
which python
which pip
python --version
pip --version

Conclusion

After reading this, you should be able to build Python 3 from source using make and create a virtual environment using your built Python.