Test Suite#

Due to its nature, build has a somewhat complex test suite, which we will try to go through in this document.

Firstly, there are two set of tests, unit tests and integration tests. In unit tests, we test the actual code implementation. In integration tests, we test build on a few real world projects; this is mostly a sanity test.

Integration tests take a long time to run, and are not very helpful tracking down issues, so they are disabled by default. They can be enabled by passing either --run-integration or --only-integration arguments to pytest, where the latter will disable the unit tests and only run the integration ones. Even though these tests are disabled by default, they will be run in CI, where test suite run durations are not a big issue.

To run the test suite we use tox, which automates running the test suite on different environments:

tox

You can find out more about how to run tox and its arguments in the tox documentation.

We have a fairly large environment matrix. We run tests for all supported Python versions and implementations, and with the module being invoked from path, sdist install, or wheel install. Additionally, we have an environment for type checking, and one to produce the documentation. There are some other extra environments, like checking the code with the minimum version of each dependency.

Some examples commands for this project:

  • Run type checking: tox -e type

  • Only run unit tests against Python 3.9: tox -e py39

  • Run both unit and integration tests: tox -- --run-integration

  • Only run integration tests: tox -- --only-integration

  • Only run integration tests with parallel tasks: tox -- -n auto --only-integration

  • Only run unit tests against Python 3.9 with the module installed via wheel: tox -e py39-wheel

We have CI testing, where we the test suite across all supported operating systems, and have test coverage reports.