April 5, 2016 · python pip

Better Package Management for Python Libraries

This post adds guidance for Python libraries to the fantastic write-up, Better Package Management.


As a maintainer of a Python library, you can break down managing dependencies into 3 concerns.

  1. How do I specify required and optional dependencies for consumers of my library?
  2. How do I specify dependencies required for contributors of my library?
  3. How do I make sure my builds are predictable and deterministic?

The answer to all three of these concerns lies in the proper usage of pip-tools.

Specifying dependencies

For Python applications that install your library, you'll need to specify the library's depencies in setup.py -

from setuptools import setup

setup(
    name='mypackage',
    install_requires=[
        'tornado>=4.2,<5',
        'thriftrw>=1.1,<2',
    ],
    extras_require={
        'thisoptionalfeature': [
            'threadloop>=1,2',
        ],
    }
)

Keep version constraints in this file as loose as possible, eg tornado>=4.2<5 instead of tornado==4.2.3; a good rule of thumb is to use Semantic Versioning, locking to the latest major release, aka mydep>=1,<2.

For library maintainers

For contributors/maintainers who are actually working on the Python library - you'll want to provide additional dependencies to enable development. Extend the deps specified in setup.py in a requirements.in file -

# extend deps in setup.py
-e .
-e .[thisoptionalfeature]

# debugging
ipdb
ipython

# linting
flake8

# releasing
wheel
zest.releaser

Now we can compile the requirements.in into a requirements.txt file like so -

(env) $ pip install pip-tools
(env) $ pip-compile

The above gives us the following fully-pinned requirements.txt -

appnope==0.1.0            # via ipython
backports-abc==0.4        # via tornado
backports.ssl-match-hostname==3.5.0.1  # via tornado
certifi==2016.2.28        # via tornado
colorama==0.3.7           # via zest.releaser
decorator==4.0.9          # via ipython, traitlets
flake8==2.5.4
gnureadline==6.3.3        # via ipython
ipdb==0.9.0
ipython-genutils==0.1.0   # via traitlets
ipython==4.1.2
mccabe==0.4.0             # via flake8
path.py==8.1.2            # via pickleshare
pep8==1.7.0               # via flake8
pexpect==4.0.1            # via ipython
pickleshare==0.6          # via ipython
pkginfo==1.2.1            # via twine
ply==3.8                  # via thriftrw
ptyprocess==0.5.1         # via pexpect
pyflakes==1.0.0           # via flake8
requests-toolbelt==0.6.0  # via twine
requests==2.9.1           # via requests-toolbelt, twine
simplegeneric==0.8.1      # via ipython
singledispatch==3.4.0.3   # via tornado
six==1.10.0               # via singledispatch, thriftrw, zest.releaser
threadloop==1.0.2
thriftrw==1.2.4
tornado==4.3              # via threadloop
traitlets==4.2.1          # via ipython
twine==1.6.5              # via zest.releaser
wheel==0.29.0
zest.releaser==6.6.4

This can be used with pip install -r requirements.txt, or with pip-sync as detailed in the following section.

Installing dependencies

Now that we have our compiled requirements.txt, we can make sure our virtualenv matches it using pip-sync -

(env) $ pip-sync

This makes sure our virtualenv contains exactly the dependencies listed in requirements.txt, and guarantees a deterministic build. Let's be nice and write a target in our Makefile as a peace-offering to future contributors -

.PHONY: install
install:
    pip install pip-tools
    pip-sync
    python setup.py develop

Which should be ran from a virtualenv like so -

$ virtualenv env
$ source env/bin/activate
(env) $ make install

Cheers.