Debian Python Policy

This document describes the packaging of Python within the Debian GNU/Linux distribution and the policy requirements for packaged Python programs and modules. This is an archive-wide policy.

Authors

License

Copyright © 1999-2015 Software in the Public Interest

This manual is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

A copy of the GNU General Public License is available as /usr/share/common-licences/GPL in the Debian GNU/Linux distribution or on the World Wide Web at The GNU General Public License.

You can also obtain it by writing to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301, USA.

On the move to Python 3

Debian currently supports two Python stacks, one for Python 2 and one for Python 3. The long term goal for Debian is to reduce this to one stack, dropping the Python 2 stack at some time.

PEP 404 states that no more major Python 2 releases are planned, although the last released major version 2.7 will see some extended support, documented in PEP 466.

Packages in Debian should use Python 3 if Python 3 is supported. New packages should use Python 3 from the initial upload, new upstream versions for existing packages should use Python 3 if the new upstream version supports it.

  • Programs should use Python 3, and should not be packaged for Python 2 as well. Python 3 should be used for the packaging if the packaging scripts use Python.
  • Python libraries should be always packaged for Python 3 if supported. Python 2 libraries should be packaged, if applications found in the reverse dependencies are not yet supported by Python 3.
  • Existing Python 2 libraries should not be dropped before the last reverse dependency is removed.

Python packaging

Versions

At any given time, the binary package python will represent the current default Debian Python version. The binary package python3 will represent the current Debian Python 3 version. As far as is reasonable, python and python3 should be treated as separate runtime systems with minimal interdependencies. In some cases, Python policy explicitly references Python helper programs such as python-support and python-central. None of these references apply to Python 3. It is a design goal to fully specify required interfaces and functions in policy for Python 3 and to avoid enshrining specific implementation details in policy. Except as noted, policy for Python 3 is the same as Python with the addition of the version number as needed to distinguish them.

The default Debian Python version should always be the latest stable upstream release that can be fully integrated in the distribution. There may be newer supported or unsupported versions included in the distribution if they are not fully integrated for a particular release.

Apart from the default version, legacy versions of Python or beta versions of future releases may be included as well in the distribution, as long as they are needed by other packages, or as long as it seems reasonable to provide them. (Note: For the scope of this document, Python versions are synonymous to feature releases, i.e. Python 2.7 and 2.7.1 are sub-minor versions of the same Python version 2.7, but Python 2.6 and 2.7 are indeed different versions.)

For any version, the main binary package must be called pythonX.Y

The set of currently supported python versions can be found in /usr/share/python/debian_defaults, the set of currently supported Python3 versions can be found in /usr/share/python3/debian_defaults. These files are in Python ConfigParser format and defines four variables in its DEFAULT section:

  • default-version which is the current default Python runtime
  • supported-versions which is the set of runtimes currently supported and for which modules should be built and byte-compiled
  • old-versions which is the list of runtimes which might still be on the system but for which should not be built anymore
  • unsupported-versions which is the list of runtimes which should not be supported at all, that is modules should not be built or byte-compiled for these. The supported interface to this file is /usr/bin/pyversions. The Python 3 interface is through /usr/bin/py3versions.

unsupported-versions is a superset of (includes) old-versions, and the default-version is always in supported-versions.

Newer versions might also appear in unsupported-versions before being moved to supported-versions.

Main packages

For every Python version provided in the distribution, the binary package pythonX.Y shall provide a complete distribution for deployment of Python scripts and applications. The package must ensure that the binary /usr/bin/pythonX.Y is provided.

Installation of pythonX.Y shall provide the modules of the upstream Python distribution with some exceptions.

Excluded are modules that cannot be included for licensing reasons (for example the profile module), for dependency tracking purposes (for example the GPL-licensed gdbm module) or that should not be included for packaging reasons (for example the tk module which depends on Xorg).

Some tools and files for the development of Python modules are split off in a separate binary package pythonX.Y-dev.

Documentation will be provided separately as well.

At any time, the python binary package must ensure that /usr/bin/python is provided as a symlink to the current pythonX.Y executable.

The python binary package must also depend on the appropriate pythonX.Y to ensure this runtime is installed.

The version of the python binary package must be greater than or equal to X.Y and smaller than X.Y+1.

Because upstream has started providing it, there will be a symlink for /usr/bin/python2 for Wheezy and later releases. See PEP 394 for details. Packages must be careful to depend on a sufficient version of python if they make use of this symlink.

Minimal packages

For every Python version provided in the distribution, the binary package pythonX.Y-minimal might exist and should not be depended upon by other packages except the Python runtime packages themselves.

Python interpreter

Interpreter name

Python scripts depending on the default Python version (see Main packages) or not depending on a specific Python version should use python (without a version) as the interpreter name.

Python scripts that only work with a specific Python version must explicitly use the versioned interpreter name (pythonX.Y).

Interpreter location

The preferred specification for the Python interpreter is /usr/bin/python or /usr/bin/pythonX.Y. This ensures that a Debian installation of Python is used and all dependencies on additional Python modules are met.

Maintainers should not override the Debian Python interpreter using /usr/bin/env python or /usr/bin/env pythonX.Y. This is not advisable as it bypasses Debian’s dependency checking and makes the package vulnerable to incomplete local installations of Python.

Module path

By default, Python modules are searched in the directories listed in the PYTHONPATH environment variable and in the sys.path Python variable. Since Python 2.4 version 2.4.5-3, Python 2.5 version 2.5.2-7, Python 2.6 version 2.6.2-1, and in all Python 2.7 versions, sys.path does not include a /usr/lib/pythonXY.zip entry anymore.

Directories with private Python modules must be absent from the sys.path.

Public Python modules not handled by python-central or python-support must be installed in the system Python modules directory, /usr/lib/pythonX.Y/dist-packages for Python 2.6 and later, and /usr/lib/pythonX.Y/site-packages for Python 2.5 and earlier. Public Python 3 modules must be installed in /usr/lib/python3/dist-packages.

Modules managed by python-support are installed in another directory which is added to the sys.path using the .pth mechanism. The .pth mechanism is documented in the Python documentation of the site module.

A special directory is dedicated to public Python modules installed by the local administrator, /usr/lib/python3/dist-packages for all Python 3 versions, /usr/local/lib/python2.Y/dist-packages for Python 2.6 and later, and /usr/local/lib/python2.Y/site-packages for Python 2.5 and earlier.

For a local installation by the administrator of Python 2.6 and later, a special directory is reserved to Python modules which should only be available to this Python, /usr/local/lib/python2.Y/site-packages (and /usr/local/lib/python3/site-packages for all Python 3 versions). Unfortunately, for Python 2.5 and earlier this directory is also visible to the system Python.

Additional information on appending site-specific paths to the module search path is available in the official documentation of the site module.

When binary packages ship identical source code for multiple Python versions, for instance /usr/lib/python2.6/dist-packages/foo.py and /usr/lib/python2.5/site-packages/foo.py, these should point to a common file. Version specific directories for identical source code are not required for Python 3 and must not be used for this.

Since Python 2.7 is the last Python 2 version and the only supported version in Wheezy and later releases, a common location to share arch-independent files across Python versions is no longer needed. Historically the location for this was /usr/share/pyshared. For Python 2.7, use of /usr/lib/python2.7/dist-packages is sufficient. For Python 3, a special location is not required, use /usr/lib/python3/dist-packages

Hooks for updates to installed runtimes

The python binary package has special hooks to allow other packages to act upon updates to the installed runtimes.

This mechanism is required to handle changes of the default Python runtime in some packages and to enable the Python packaging helpers.

There are three supported hook types which come in the form of scripts which are invoked from the maintainer scripts of the Python runtime packages when specific installations, removals, or upgrades occur.

  • /usr/share/python/runtime.d/\*.rtinstall: these are called when a runtime is installed or becomes supported. The first argument is “rtinstall”, the second argument is the affected runtime (for example pythonX.Y) and the third and fourth argument are the old and new version of this packaged runtime if this runtime was already installed but unsupported.
  • /usr/share/python/runtime.d/\*.rtremove: these are called when a runtime is removed or stops being supported. The first argument is “rtremove”, and the second argument is the affected runtime (for example pythonX.Y).
  • /usr/share/python/runtime.d/\*.rtupdate: these are called when the default runtime changes. The first argument is either “pre-rtupdate”, called before changing the default runtime, or “rtupdate”, called when changing the default runtime, or “post-rtupdate”, called immediately afterwards. The second argument is the old default runtime (for example pythonX.Y), and the third argument is the new default runtime (for example pythonX.Z).

Documentation

Python documentation is split out in separate binary packages pythonX.Y-doc. The binary package python-doc will always provide the documentation for the default Debian Python version.

TODO: Policy for documentation of third party packages.

Packaged modules

The goal of these policies is to reduce the work necessary for Python transitions. Python modules are internally very dependent on a specific Python version. However, we want to automate recompiling modules when possible, either during the upgrade itself (re-byte-compiling pyc and pyo files) or shortly thereafter with automated rebuilds (to handle C extensions). These policies encourage automated dependency generation and loose version bounds whenever possible.

Types of Python modules

There are two kinds of Python modules, “pure” Python modules, and extension modules. Pure Python modules are Python source code that generally works across many versions of Python. Extensions are C code compiled and linked against a specific version of the python runtime, and so can only be used by one version of Python.

Some distributions link extensions to libpython, but this is not the case in Debian as symbols might as well be resolved by /usr/bin/pythonX.Y which is not linked to libpython.

Python packages are a way of structuring Python’s module namespace by using “dotted module names”. See Python’s documentation for details on how packages are defined in Python terms (a package in the Python sense is unrelated to a Debian package). Python packages must be packaged into the same directory (as done by upstream). Splitting components of a package across directories changes the import order and may confuse documentation tools and IDEs.

There are two ways to distribute Python modules. Public modules are installed in a public directory as listed in Module path. They are accessible to any program. Private modules are installed in a private directory such as /usr/share/package-name or /usr/lib/package-name. They are generally only accessible to a specific program or suite of programs included in the same package.

Wheels

PEP 427 defines a built-package format called “wheels”, which is a zip format archive containing Python code and a “dist-info” metadata directory, in a single file named with the .whl suffix. As zip files, wheels containing pure-Python can be put on sys.path and modules in the wheel can be imported directly by Python’s “import” statement. (Importing extension modules from wheels is not yet supported as of Python 3.4.)

Except as described below, packages must not build or provide wheels. They are redundant to the established way of providing Python libraries to Debian users, take no advantage of distro-based tools, and are less convenient to use. E.g. they must be explicitly added to sys.path, cannot be easily grepped, and stack traces through zips are more difficult to debug.

A very limited set of wheel packages are available in the archive, but these support the narrow purpose of enabling the pip tool, in a Debian policy compliant way. The set of packages providing wheels for this purpose are (by source package name):

  • chardet
  • distlib
  • html5lib
  • python-colorama
  • python-pip
  • python-setuptools
  • python-urllib3
  • requests
  • six

Wheel packages supporting pyvenv and pip are named with the python- prefix, and the -whl suffix, e.g. python-chardet-whl. When these binary packages are installed, their .whl files must be placed in the /usr/share/python-wheels directory. Such wheels must be built with the --universal flag so as to generate wheels compatible with both Python 2 and Python 3.

Module package names

Public modules used by other packages must have their binary package name prefixed with python-. It is recommended to use this prefix for all packages with public modules as they may be used by other packages in the future. Python 3 modules must be in a separate binary package prefixed with python3- to preserve run time separation between Python 2 and Python 3.

The binary package for module foo should preferably be named python-foo, if the module name allows, but this is not required if the binary package ships multiple modules. In the latter case the maintainer chooses the name of the module which represents the package the most.

For subpackages such as foo.bar, the recommendation is to name the binary packages python-foo.bar and python3-foo.bar.

Such a package should support the current Debian Python version, and more if possible (there are several tools to help implement this, see Packaging tools). For example, if Python 2.5, 2.6, and 2.7 are supported, the Python statement:

import foo

should import the module when the user is running any of /usr/bin/python2.5, /usr/bin/python2.6, and /usr/bin/python2.7. This requirement also applies to extension modules; binaries for all the supported Python versions should be included in a single package.

Specifying supported versions

The optional X-Python-Version (preferred) or XS-Python-Version field in the general paragraph (the first one, for the source package) of debian/control specifies the versions of Python (not versions of Python 3) supported by the source package. Similarly, X-Python3-Version is used to specify the versions of Python 3 supported by the package. When not specified, they default to all currently supported Python (or Python 3) versions.

They are used by some packaging scripts to automatically generate appropriate Depends and Provides lines. The format of the field may be one of the following:

X-Python-Version: >= X.Y
X-Python-Version: >= A.B, << X.Y
XS-Python-Version: A.B, X.Y
XS-Python-Version: all

The keyword “all” means that the package supports any Python version available but might be deprecated in the future since using version numbers is clearer than “all” and encodes more information. The keyword “all” is limited to Python versions and must be ignored for Python 3 versions. Lists of multiple individual versions (e.g. 2.4, 2.5, 2.6) work for XS-Python-Version and will continue to be supported, but are not recommended and are not supported by X-Python-Version or X-Python3-Version for Wheezy and later releases.

The keyword “current” has been deprecated and used to mean that the package would only have to support a single version (even across default version changes). It must be ignored for Python 3 versions.

The use of XB-Python-Version in the binary package paragraphs of debian/control file has been deprecated and should be removed in the normal course of package updates. It never achieved sufficient deployment to support it’s intended purpose of managing Python transitions. This can be adequately accomplished by examining package dependencies.

Dependencies

Packaged modules available for the default Python version (or many versions including the default) as described in Module package names must depend on python (>= X.Y). If they require other modules to work, they must depend on the corresponding python-foo. They must not depend on any pythonX.Y-foo.

Packaged modules available for one particular version of Python must depend on the corresponding pythonX.Y package instead. If they need other modules, they must depend on the corresponding pythonX.Y-foo packages, and must not depend on any python-foo.

Provides

Provides in binary packages of the form pythonX.Y-foo must be specified if the package contains an extension for more than one Python version and another package with version specific dependencies on the package require it. Provides are only for extensions, not modules. Provides should only be rarely used for Python packages and never for Python 3.

Modules byte-compilation

If a binary package provides any binary-independent modules (foo.py files), the corresponding byte-compiled modules (foo.pyc files) and optimized modules (foo.pyo files) must not ship in the package. Instead, they should be generated in the package’s postinst, and removed in the package’s prerm. The package’s prerm has to make sure that both foo.pyc and foo.pyo are removed.

A binary package should only byte-compile the files which belong to the package.

The file /etc/python/debian_config allows configuration for how modules should be byte-compiled. The postinst scripts should respect these settings.

Pure Python modules in private installation directories that are byte-compiled with the default Python version must be forcefully byte-compiled again when the default Python version changes.

Public Python extensions should be bin-NMUed.

Private Python extensions should be subject to binary NMUs every time the default interpreter changes, unless the extension is updated through a .rtupdate script.

Python programs

Programs using the default python

Programs that can run with any version of Python must begin with #!/usr/bin/python or #!/usr/bin/env python (the former is strongly preferred). They must also specify a dependency on python, with a versioned dependency if necessary.

If the program needs the python module foo, it must depend on the real package providing this module, usually python-foo but this name might vary when the package ships multiple modules.

Program shipping private modules

A program using /usr/bin/python as the interpreter can come up with private Python modules. These modules should be installed in /usr/share/<module>, or ``/usr/lib/<module> if the modules are architecture-dependent (e.g. extensions).

The rules explained in Modules byte-compilation apply to those private modules: the byte-compiled modules must not be shipped with the binary package, they should be generated in the package’s postinst, using the current default Python version, and removed in the prerm. Modules should be byte-compiled using the current default Python version.

Programs that have private compiled extensions must either handle multiple version support themselves, or declare a tight dependency on the current Python version (e.g. Depends: python (>= 2.6), python (<< 2.7)). No tools currently exist to alleviate this situation.

Programs using a particular Python version

A program which requires a specific version of Python must begin with #!/usr/bin/pythonX.Y (or #!/usr/bin/env pythonX.Y). It must also specify a dependency on pythonX.Y and on any pythonX.Y-foo package providing necessary modules. It should not depend on any python-foo package, unless it requires a specific version of the package (since virtual packages cannot be versioned). If this is the case, it should depend on both the virtual package and the main package (e.g. Depends: python2.7-foo, python-foo (>= 1.0)).

The notes on installation directories and byte-compilation for programs that support any version of Python also apply to programs supporting only a single Python version. Modules to be byte-compiled should use the same Python version as the package itself.

Package names for programs

Binary packages that provide executable programs should be named differently than module packages. Specifically, such binary packages should not start with either python- or python3- as this is not supported by the Python helper scripts for this purpose. Generally, if a single executable is provided, name the package that contains it after the executable’s name. If that is too generic, or clashes with existing package names, or the package contains multiple executables, use something like foo-cli, foo-bin or even foo-python.

Programs embedding Python

Building embedded programs

Programs which embed a Python interpreter must declare a Build-Depends on pythonX.Y-dev, where pythonX.Y is the Python version the program builds against. It should be the current default Python version unless the program does not work correctly with this version.

Embedded Python dependencies

Dependencies for programs linking against the shared Python library will be automatically created by dpkg-shlibdeps. The libpythonX.Y.so.Z library the program is built against is provided by the pythonX.Y package.

Interaction with locally installed Python versions

As long as you don’t install other versions of Python in your path, Debian’s Python versions won’t be affected by a new version.

If you install a different sub-release of the version of Python you have got installed, you will need to be careful to install all the modules you use for that version of Python too.

Build dependencies

Build dependencies for Python dependent packages must be declared for every Python version that the package is built for. The python-all-dev should be used when building extensions for any or all Python versions. To build for a specific version or versions, Build-Depend on pythonX.Y-dev.

Some applications and pure Python modules may be able to build-depend only on python or python-all and not require the -dev packages. Packages that do not require the -dev packages must not Build-Depend on them.

Build-Depend on at least:

Build-Depends: python2.7
Build-Depends: python2.6 (>= 2.6-1)
Build-Depends: python (>= 2.6.6-9)
Build-Depends: python-all

Build-Depends: python2.7-dev
Build-Depends: python2.6-dev (>= 2.6-1)
Build-Depends: python-dev (>= 2.6.6-9)
Build-Depends: python-all-dev
Build-Depends: python3-all-dev (>= 3.2)

If you use either python-support or python-central you must additionally Build-Depend on those.

Packaging tools

This section describes the various tools to help package Python programs and modules for Debian. Although none of these tools are mandatory, their use is strongly encouraged, as the above policy has been designed with them in mind (and vice versa). This section is just an overview. If you use these tools, you should read their full documentation.

distutils

The standard Python distutils module has been modified in Debian to change the default installation directory of public Python modules and to add a new flag to the “install” command to override the default, --install-layout=.

To allow the use this flag, maintainers should ensure that at least version 2.6.2-1 will be used for Python 2.6, version 2.5.4-1 for Python 2.5, and version 2.4.6-2 for Python 2.4. This flag is parsed but ignored in Python 2.4 and Python 2.5.

Public Python modules installed with a modified distutils default to /usr/local/lib/pythonX.Ydist-packages for Python 2.6 and later. This directory is seen by the system-provided Python 2.6.

When using a system-provided Python 2.4 or Python 2.5, the default is /usr/lib/pythonX.Y/site-packages which is seen by the system-provided Python 2.4 and Python 2.5 versions, but not by a system-provided Python 2.6 and later versions.

When using a local Python installation, the default is /usr/local/lib/pythonX.Y/site-packages which is only seen by the local Python installation.

Using the --install-layout=deb flag to the “install” command of setup.py with a system-provided Python 2.6 or later versions, Python modules will be installed to /usr/lib/pythonX.Y/dist-packages which is only seen by the system-provided Python, not by a local installation.

Using the --install-layout=deb flag to setup.py with a system-provided Python 2.4 or Python 2.5 does not affect the default installation directory.

python-support (deprecated)

python-support is deprecated. It is a system intended to provide a simple way to byte-compile pure Python modules and manage dependencies. It integrates with debhelper, manages byte-compilation, private modules, integrates with runtime update hooks, and will fill-in the ${python:Depends}, ${python:Versions}, and ${python:Provides} substvars.

See the python-support documentation in /usr/share/doc/python-support for details.

pycentral (removed)

python-central has been removed from Jessie and later releases. It provided another way to manage Python modules.

CDBS

The CDBS python-distutils.mk class helps packaging of setup.py-based Python packages.

dh_python2 and dh_python3

dh_python2 and dh_python3 are debhelper extensions provided as part of Python 2 and Python 3 to make it easier to package Python modules and extensions. They calculate Python dependencies, add maintainer scripts to byte compile files, etc. Their use is not mandatory, but they are recommended by the Python maintainers.

See man dh_python2 or man dh_python3 for details.

Upgrade procedure

This section describes the procedure for the upgrade when the default python version is changed in the unstable distribution, requiring recompilation of many python-related packages.

  • Selected pre-releases and release candidates of new Python versions are uploaded to Experimental to support pre-transition work and testing.

  • Application and module maintainers make sourceful changes where needed to prepare for the new Python version when needed.

  • Have a long and heated discussion.

  • The Debian Python maintainer and module/application maintainers discuss the

    readiness for a new default Debian Python version and associated packaging/policy changes. Once there is some consensus, the Python maintainer announces the upgrade and uploads to Unstable.

  • Upload of the Python core meta-packages python, python-dev, python-doc, and several python-<module>, depending on the new `pythonX.Y, pythonX.Y-dev and so on.

  • The release team schedules rebuilds for packages that may need it. Packages that require additional manual work get updated and uploaded.