Python Packaging Tutorial

Submitted by nanodano on Tue, 12/14/2021 - 10:53

Introduction

This tutorial will cover setuptools and how to package libraries and applications for pip, https://pypi.org, and distribution to others.

These examples will show you how to create distributable packages. We'll look at how import works, how the setup.py file and how to use it to build and install packages. I recommend creating a new virtual environment and activating it before testing out any of these examples. If you aren't familiar, read up on virtual environments. That will ensure you have an isolated environment do not contaminate your system install or any project. You can also get started with a simple distutils example from the official documentation. See my Python Virtual Environments Tutorial if you need to learn more about virtual environments.

Also see Python import, sys.path, and PYTHONPATH Tutorial and my Python Virtual Environment Tutorial.

If you need Python, check out my tutorials:

Terminology

Before going any further, let's look at some terms that might be confusing:

Module - A Python file (.py) with code, functions, classes, etc.
Package - A directory with Python modules files. Might have an init.py and/or main.py along with Python modules (.py files). The word package also commonly refers to a distributable package.
Distributable package - A Python library or application that was pacakged with a setup.py file that is built in to a distributable file that can be published on PyPI.org. This term may be shortened to just "package" which can cause confusion between the other use of the word package which refers to a directory with Python modules. Even in this tutorial, I may refer to a distribution as a "package" and when dicussing "packaging" it refers to creating a distributable.
setuptools - The python package used to create and manage distributable Python packages.
distutils - The older version of setuptools, which is now legacy. Use setuptools instead.
setup.py - The primary file of a distributable package that defines author, version information, files to include, etc. Used to build, install, and upload the package it belongs to. Builds packages that can be installed using pip. Functions provided by distutils package.
pip - Tool for downloading, installing, uninstalling, upgrading distirubted packages in a Python instance. Works with packages built using setuptools and setup.py.
PyPI - Python Package Index, pypi.org - the public repository for Python packages.

Minimal package

This is the simplest possible package that you can build and install. I would never recommend making a package like this, but it demonstrates the core element of a distributable package, the setup.py file.

This example will default to version 0.0.0, have no author information, and in general, is not fit for sharing. Technically it will work and you can install modules to your local site-packages for re-use but again, this is only an example for learning. Read the next section for how to make a better package.

All you really need to do to turn your regular package in to a package ready for distribution is add a setup.py file. The setup.py file defines the package name, version number, author information, files, and any additional setup. All of the magic comes with this setup.py file. This is a really simple setup.py example, taken from, https://docs.python.org/3.7/distutils/introduction.html#distutils-simple-example. Other useful resources include an example pypi project or the setup script documentation.

Note that if you just want to create a package for use locally in a single project, you do not need a setup.py and do not need to create a distributable package. This describes the process for building a package that is intended to be installed to your Python site-packages for use among multiple projects or shared with other people.

# Directory structure
\              # Project directory root
\setup.py      # Describes distributable package info
\mylib.py      # Contains the python code for sharing

# setup.py
from setuptools import setup

setup(
  name='my_module',
  py_modules=['my_module'],  # Include my_module.py
)

The example above shows the simples possible distributable package you can make. It may not make it up to pypi.org but it will technically install locally. All you need is a single .py file (your module) and a setup.py to provide the name and list of modules to include. With this, you can run python setup.py install and it will install the module to your site-packages directory.

python setup.py install

When you install it, the package will be copied in to your site packages directory making your package available for import. To learn more about import, sys.path, PYTHONPATH and how Python searches for modules to import, check out my Python import, sys.path, and PYTHONPATH Tutorial .

Then you would import it like this in a Python script:

import my_module

Install versus develop

python setup.py install
# or
python setup.py develop 
# develop will create symlinks so it doesn't require reinstall after every change
# This works in Windows too!

See other setup.py commands with:

python setup.py --help-commands

Create a distribution package

There are different distribution options. The simplest option is the source distribution (sdist) where it simply packages up all the source code.

Other packages can be built using a build distribution (bdist) which are good for packages with binary blobs or creating a special format.

Some options include rpm and msi for Linux packages and Windows installers!

pip install twine # tool to push to pypi
python setup.py sdist  # Package the release file
twine upload dist/<package_file>

# Other package options
python setup.py bdist --format=gztar # .tar.gz
python setup.py bdist --format=zip # .zip
python setup.py bdist --format=rpm # Yum package (Fedora/CentOS/RHEL)
python setup.py bdist --format=msi # Windows installer
python setup.py bdist --format=wininst # self extracting zip

Push package to PyPI.org

To update your package and push it to pypi, update your setup.py to make the version number higher than the current version, and repackage with python setup.py sdist. Follow that up with a twine upload.

# Build the package
python setup.py sdist
# Upload the package
twine upload dist/mypackage.tar.gz

Set up \~/.pypirc

When using twine to upload, for example, twine upload dist/mypackage-1.0.0.tar.gz, you can avoid entering your password each time by setting up a ~/.pypirc file.

This step is optional, but if you upload packages to pypi.org regularly, this will save a lot of time.

Here is an example:

# ~/.pypirc

[distutils]
index-servers=pypi

[pypi]
# repository = https://upload.pypi.org/legacy/
username = myusername
password = mysecretpassword

Name the file .pypirc and place it in your $HOME directory.

Create a distributable package

This will explain how to package a Python application for sharing on pypi.org that can be installed with pip. Another method of packaging an application is to create a standalone .exe, .app, or Linux executable. You can use PyInstaller to do this. If you are interested in that, check out my PyInstaller tutorial.

The previous example is an absolute minimum. It didn't have any author or version information about the module. You can provide a lot more useful information in the setup.py file. Not only can you provide links to a website or documentation, you can specify executable scripts that should be installed with your module and dependencies that it relies on. This example is a better template for a new package.

# Directory structure
\
\README.rst             # Simple documentation
\setup.py               # Used to build package and install
\mypackage\__init__.py  # Executed when package is imported (optional)
\mypackage\__main__.py  # Executed with python -m mypackage (optional)
\mypackage\mymodule1.py # Modules the package provides/uses
\mypackage\mymodule2.py # Modules the package provides/uses

That's the core of a basic package. You do not have to create a package and can simply provide modules as demonstarted in the first example. In the setup.py file you have options for py_modules to include and packages to include. For more information about what you can provide in the setup.py file check out the official documentation on the setup script.

Add executable scripts

Option 1) Add scripts Option 2) Use entrypoint

When addding executable scripts to a package, I recommend placing them in a bin/ directory. Then, in the setup.py file, provide the list of scripts to the scripts parameter.

. Create a .bat file for Windows and a shell script with a shebang for Linux/Mac. Refer to the examples provided for how to build the scripts.

Entrypoints (scripts)

From https://setuptools.readthedocs.io/en/latest/setuptools.html#automatic-script-creation

Packaging and installing scripts can be a bit awkward with the distutils. For one thing, there’s no easy way to have a script’s filename match local conventions on both Windows and POSIX platforms. For another, you often have to create a separate file just for the “main” script, when your actual “main” is a function in a module somewhere. And even in Python 2.4, using the -m option only works for actual .py files that aren’t installed in a package.

setuptools fixes all of these problems by automatically generating scripts for you with the correct extension, and on Windows it will even create an .exe file so that users don’t have to change their PATHEXT settings. The way to use this feature is to define “entry points” in your setup script that indicate what function the generated script should import and run. For example, to create two console scripts called foo and bar, and a GUI script called baz, you might do something like this:

setup(
    # other arguments here...
    entry_points={
        'console_scripts': [
            'foo = my_package.some_module:main_func',
            'bar = other_module:some_func',
        ],
        'gui_scripts': [
            'baz = my_package_gui:start_func',
        ]
    }
)

PyInstaller is another option for packaging applications for distribution. PyInstaller lets you create Windows .exe, Mac .app, and Linux executable files. You can even have it package as a single file. You can follow my PyInstaller tutorial here: https://www.devdungeon.com/content/pyinstaller-tutorial.

scripts vs entry_points

Example setup.py template

Here is an example setup.py that you can use a starting place for your own package. Remove any pieces you don't need and modify everything else.

# setup.py template
# https://docs.python.org/3/distutils/setupscript.html
import os
from setuptools import setup
from sys import platform

requirements = [
    'docopt',
    'discord.py==0.16.12'
]

if platform == "win32":
    requirements.append('windows-curses')

setup(
    name = "an_example_pypi_project",
    version = "0.0.4",
    author = "Andrew Carter",
    author_email = "andrewjcarter@gmail.com",
    description = ("An demonstration of how to create, document, and publish "
                   "to the cheese shop a5 pypi.org."),
    license = "MIT",
    keywords = "example documentation tutorial",
    url = "http://packages.python.org/an_example_pypi_project",

    # Individual Python modules (.py files)
    py_modules=['mymodule'],
    # Directories (package)
    packages=[
        'mypackage',
        'tests'
    ],
    long_description_content_type='text/markdown',
    long_description=open('README.md').read(),
    scripts=scripts,
    package_data={  # Other misc files that should be installed
        'mypackage': [ 
            'data/defaults.xml',  # Relative to setup.py dir
            'data/*.dat',
        ],
    },
    python_requires='<3.7',  # Optional
    install_requires=requirements,
    platforms='any',
    classifiers=[  # Options at https://pypi.org/classifiers/
        'Development Status :: 4 - Beta',
        'Environment :: Console :: Curses',
        'Environment :: Web Environment',
        'Environment :: MacOS X',
        'Environment :: X11 Applications :: Qt',
        'Framework :: Django :: 2.1',
        'Framework :: Flask',
        'Intended Audience :: End Users/Desktop',
        'Intended Audience :: Developers',
        'Intended Audience :: System Administrators',
        'Natural Language :: English',
        'License :: OSI Approved :: MIT License',
        'Operating System :: Android',
        'Operating System :: MacOS :: MacOS X',
        'Operating System :: Microsoft :: Windows',
        'Operating System :: POSIX',
        'Programming Language :: Python',
        'Topic :: Utilities',
        'Topic :: Software Development',
        'Topic :: Security',
    ],
)

You can also add dependencies from GitHub.

Create a standalone .exe, .app, or Linux Executable

If you want to package your application as a standalone executable, I recommend using PyInstaller. This allows you to easily distribute your app without the user needing to install or even know anything about Python. Check out my PyInstaller Tutorial for more information.

Conclusion

After reading this, you should understand how to create distributable packages with distutils using setup.py. You should be able to create your own package, add dependencies, include executable scripts, build it, and upload it to pypi.org for others to download.