April 5, 2016 · python pip

Better Package Management for Python Libraries

This post adds guidance for Python libraries to the fantastic write-up, Better Package Management.


As a maintainer of a Python library, you can break down managing dependencies into 3 concerns.

  1. How do I specify required and optional dependencies for consumers of my library?
  2. How do I specify dependencies required for contributors of my library?
  3. How do I make sure my builds are predictable and deterministic?

The answer to all three of these concerns lies in the proper usage of pip-tools.

Specifying dependencies

For Python applications that install your library, you'll need to specify the library's depencies in setup.py -

from setuptools import setup

setup(  
    name='mypackage',
    install_requires=[
        'tornado>=4.2,<5',
        'thriftrw>=1.1,<2',
    ],
    extras_require={
        'thisoptionalfeature': [
            'threadloop>=1,2',
        ],
    }
)

Keep version constraints in this file as loose as possible, eg tornado>=4.2<5 instead of tornado==4.2.3; a good rule of thumb is to use Semantic Versioning, locking to the latest major release, aka mydep>=1,<2.

For library maintainers

For contributors/maintainers who are actually working on the Python library - you'll want to provide additional dependencies to enable development. Extend the deps specified in setup.py in a requirements.in file -

# extend deps in setup.py
-e .
-e .[thisoptionalfeature]

# debugging
ipdb  
ipython

# linting
flake8

# releasing
wheel  
zest.releaser  

Now we can compile the requirements.in into a requirements.txt file like so -

(env) $ pip install pip-tools
(env) $ pip-compile

The above gives us the following fully-pinned requirements.txt -

appnope==0.1.0            # via ipython  
backports-abc==0.4        # via tornado  
backports.ssl-match-hostname==3.5.0.1  # via tornado  
certifi==2016.2.28        # via tornado  
colorama==0.3.7           # via zest.releaser  
decorator==4.0.9          # via ipython, traitlets  
flake8==2.5.4  
gnureadline==6.3.3        # via ipython  
ipdb==0.9.0  
ipython-genutils==0.1.0   # via traitlets  
ipython==4.1.2  
mccabe==0.4.0             # via flake8  
path.py==8.1.2            # via pickleshare  
pep8==1.7.0               # via flake8  
pexpect==4.0.1            # via ipython  
pickleshare==0.6          # via ipython  
pkginfo==1.2.1            # via twine  
ply==3.8                  # via thriftrw  
ptyprocess==0.5.1         # via pexpect  
pyflakes==1.0.0           # via flake8  
requests-toolbelt==0.6.0  # via twine  
requests==2.9.1           # via requests-toolbelt, twine  
simplegeneric==0.8.1      # via ipython  
singledispatch==3.4.0.3   # via tornado  
six==1.10.0               # via singledispatch, thriftrw, zest.releaser  
threadloop==1.0.2  
thriftrw==1.2.4  
tornado==4.3              # via threadloop  
traitlets==4.2.1          # via ipython  
twine==1.6.5              # via zest.releaser  
wheel==0.29.0  
zest.releaser==6.6.4

This can be used with pip install -r requirements.txt, or with pip-sync as detailed in the following section.

Installing dependencies

Now that we have our compiled requirements.txt, we can make sure our virtualenv matches it using pip-sync -

(env) $ pip-sync

This makes sure our virtualenv contains exactly the dependencies listed in requirements.txt, and guarantees a deterministic build. Let's be nice and write a target in our Makefile as a peace-offering to future contributors -

.PHONY: install
install:  
    pip install pip-tools
    pip-sync
    python setup.py develop

Which should be ran from a virtualenv like so -

$ virtualenv env
$ source env/bin/activate
(env) $ make install

Cheers.