Running Commands with Tox¶
Tox is a general purpose tool for automating Python testing. We recommend using tox to specify the environments in which your tests are run, both locally and on Continuous Integration services.
Getting Started with Tox: Running tests¶
The first thing to configure tox to do is to run the tests for a package. The most minimal tox file for a package following this guide is:
[tox]
envlist = py38
isolated_build = True
[testenv]
extras = test
commands = pytest {posargs}
Let’s dig into the sections of this file, the [tox]
section is the global
configuration for
the whole file. We use this to define envlist
which is a list of all the
different builds configured in tox, here we set this to be a Python 3.8
environment, we will expand on this shortly. The isolated_build
configuration option configures tox to build your source distribution in the
same manner as recommended in Releasing Your Package.
The [testenv]
section describes settings common to all environments you
specify in the tox file (unless they are later overridden), here we default
the commands =
option to run pytest. The {posargs}
is a tox
substitution which
passes extra arguments through to pytest
.
The extras = test
line tells tox to install the optional-dependencies
section
listed in pyproject.toml
for running your test suite; this should include pytest
.
To run your tests with tox run:
$ tox -e py38
To pass arguments through to pytest
use --
here we tell pytest to
stop after the first failure.
$ tox -e py38 -- -x
Multiple builds¶
Tox allows configuration of multiple builds in a few different ways, the easiest one is to specify multiple Python versions in the env list:
[tox]
envlist = py{37,38}
isolated_build = True
This takes our one test configuration and makes a Python 3.7 and a Python 3.8 environment that can be seen by listing all tox environments with:
$ tox -l
py38
py37
This feature is called generative envlist and can be used to create many build environments with minimal repetition.
Named Environments¶
Using generative build environments you can define extra named environments
which can be useful for builds that need to specify specific dependencies or
settings. So far on this page we have assumed that all your dependencies are
specified in pyproject.toml. You can extend or override this by using the
deps =
configuration option in tox. Here we define a named test
environment which installs the development version of numpy.
[tox]
envlist = py{37,38}{-numpydev,}
isolated_build = True
[testenv]
extras = test
commands = pytest {posargs}
deps =
numpydev: git+https://github.com/numpy/numpy
the envlist
is now more complex, the result of this the following:
$ tox -l
py37-numpydev
py37
py38-numpydev
py38
with the deps
overridden for numpydev
builds.
Environment variables¶
It is often useful to set environment variables within the building and testing
environment prior to testing. Environment variables can be set within tox.ini
with:
[testenv]
# Pass through the following environment variables which may be needed for the CI
passenv = HOME, WINDIR, LC_ALL, LC_CTYPE, CC, CI, TRAVIS
# Suppress display of matplotlib plots generated during docs build
setenv = MPLBACKEND=agg
The variables listed after passenv
will be preserved from the
environment that you used to run tox, while the setenv
variables
are set within the testing environment. In the template, we have set the
MPLBACKEND
variable to the agg
backend, which prevents matplotlib
from launching interactive plot displays when generating figures from the
matplotlib plot directive or pytest-mpl. For more on making use of this
feature, see Add plots to your documentation.
Building Documentation with tox¶
One common task which isn’t running the test suite is building sphinx
documentation, documentation builds can be complex with a number of extra
dependencies or settings. In this section we will add a build_docs
named
environment to tox. This section assumes you have already followed
Documenting your Package.
[testenv:build_docs]
extras = docs
commands = sphinx-build docs docs/_build/html -W -b html {posargs}
This section installs the package extras for the documentation, which should
be a list of all your documentation dependencies and then sets the command to
be the sphinx-build command to
build the docs and output them in the docs/_build/html
folder relative to
the tox.ini
file.
You can now run your documentation with:
$ tox -e build_docs
you can pass through extra arguments to sphinx-build because of
the {posargs}
substitution. For example to force sphinx to ignore its
cache you can run:
$ tox -e build_docs -- -aE
Testing Packages with Compiled Extensions¶
As configured in this guide so far, tox will perform the following actions (all in the same directory as the tox.ini
file):
python setup.py sdist
Create a new virtualenv
Install the built sdist.
Run the commands listed in
commands =
, which here we assume to bepytest
.
(See https://tox.readthedocs.io/en/latest/index.html#system-overview for more details.)
For packages laid out as described in this guide, i.e. with the Python
package in a directory in the root repo, i.e. astropy/
, this means that
when pytest
is run, it will collect the tests from the local directory
(as desired), and all imports of the package i.e. astropy
will be
imported from the local directory not the installed sdist.
For pure python packages this generally isn’t a problem, the contents of the installed sdist and the local directory are the same (tox just made the sdist from the local directory). However, for packages that include compiled extensions, the installed package and the local directory are not the same. The installed package has build the compiled extensions, and the local directory does not. This means that unless you make some adjustments to the package or the tox configuration compiled extensions will not work when running pytest through tox as described above.
There are two main ways to alleviate this issue:
1. Move the Python package source code under a src/
folder in the root of
the repo. This is a common package layout for Python projects, and it means
that you can not import your package relative to the git root, meaning it
will be imported from the installed sdist, see https://setuptools.readthedocs.io/en/latest/setuptools.html#using-a-src-layout for details.
2. Configure tox to run pytest
from a temporary directory so that the
local import does not work. With this method you make use of pytest’s
–pyargs flag
to run the tests against the installed version of the package. This ensures
that any compiled extensions are properly detected, but prevents things like
specifying paths to pytest from working.
To configure tox to run pytest
from a temporary directory do the
following in tox.ini
:
[tox]
envlist = py38
isolated_build = True
[testenv]
changedir = tmp
extras = test
commands = pytest --pyargs packagename {posargs}
replacing packagename
with the name of your package as you import it,
i.e. astropy
.