When time has come, we may want to publish a python project as a package onto PyPI. While a lot of my work were proprietary, there is finally a good project that I can publish and this is what to do to put it up:

There is a site test.pypi.org for testing. We should use that and check before we make the real one. First we have to register an account on test.pypi.org and after that, generate a token so we can use the token for authentication instead of username and password when we publish the package.

Once we did that we have to create $HOME/.pypirc with the following:

[distutils]
index-servers=
    pypi
    testpypi

[testpypi]
repository: https://test.pypi.org/legacy/
username: __token__
password: pypi-AgE.....

[pypi]
username: __token__
password: pypi-AgE...

If we use token, we have to say username: __token__ and put the token as password. Similarly for the real repository, pypi.org as the two sites do not share the same authentication system. Hence we have the .pypirc as above.

The next step is to clean up the project. We can see how other projects doing this. In general, we need:

  • README
  • LICENSE
  • MANIFEST.in
  • changelog
  • requirements.txt
  • setup.py

Not all of these are required but it is good to have. README and changelog is quite free-style. I would prefer to do it in markdown. You may use keep a changelog as a guideline. LICENSE should be created and provided for you when you create your project on github, for example. The file requirements.txt is quite standard in python projects and helps pip to install dependent packages. An example of it is as follows:

lxml>=3.7.3
requests==2.20.0

Or you can skip the version and just put the name of the package there. MANIFEST.in tells what to include in the python package. Usually it is a list of non-code files. An example is as follows, as we are required to include the license to fulfil the requirement of GPL:

include LICENSE
include README.md

The most complicated one is setup.py. I will keep my version as a template and use it every time I make a new project. Here is how it looks like:

import os.path
from setuptools import setup, find_packages

CWD = os.path.abspath(os.path.dirname(__file__))

# Get the long description from the README file
with open(os.path.join(CWD, 'README.md'), encoding='utf-8') as f:
    long_description = f.read()

import my_package

packages = find_packages(exclude=['contrib', 'docs', 'tests'])  # py module name
package_data = {}
requires = []
classifiers = [
	# See https://pypi.org/classifiers/
    "Development Status :: 3 - Alpha",
    "Programming Language :: Python :: 3",
]

setup(
    name = 'my_package',  # need a weird name to prevent conflict
    description = 'blah blah'
    version = my_package.__version__,
    author = my_package.__author__,
    author_email = my_package.__author_email__,
    long_description = long_description,
    long_description_content_type = 'text/markdown',

    # Look for package directories automatically
    packages = packages,
	py_modules = ['my_package'],
    package_data = package_data,

    # runtime dependencies
    install_requires = requires,
    url = "https://example.com",
    license = "GPL",
    classifiers = classifiers,
)

Most of the key things are included above. The keyword argument install_requires to setup() should be a list of package names that we need to install, packages argument should be list of all python files of this project, to be discovered by the find_packages() call. But if the module is a flat hierarchy (i.e., the code is not inside a directory of the same name but as a .py file at the top level), we have to specify it as py_modules argument, with list of names of the module (no .py extension). Of course, we need the name, author, version, etc. to publish onto pypi.org and they are part of the module-level constants defined in the code.

Once we finish with the setup.py we can test locally by running the following command at the project’s top directory:

pip install .

Once this works, we can build the package. A compiled package may be more complicated but the flow should be the same. First we build a source distribution package (i.e., a tarball of all codes and files of the module) as well as a wheel package of it (for convenience of the users only, not mandatory to produce both):

python setup.py sdist
python setup.py bdist_wheel

Then a tarball and a .whl file will be created in dist/ directory under the project’s root dir. We can further test it by pip install the corresponding local file.

Then we register and upload the package: We first pip install the tool twine and then upload everything in dist:

twine upload -r testpypi dist/*

then wait a few minutes for pypi to upload the index and run the following to ensure it works correctly:

pip install --index-url https://test.pypi.org/simple/ your-package

If everything’s fine, we can then repeat the above command on the real pypi.org:

twine upload dist/*
pip install your-package

We do not need additional arguments to the command as pypi is the default index to use

Reference

https://packaging.python.org/guides/using-testpypi/