Continuous Integration

Overall Strategy

Currently, we use CircleCI to manage continuous integration for pyani development. We use CodeCov to monitor test coverage and CodeFactor for automated code review.

Tests are managed using the pytest package, and the pytest-cov plugin to generate test coverage output.

Tests are run on demand for GitHub pull requests at the pyani repository, and weekly for the master branch.

The CircleCI config file is found at .circleci/config.yml under the repository root.

Operating System and Python versions

We use pre-built CircleCI Docker convenience images as a base for testing. These are all based on Ubuntu. We use distinct images for each of the Python versions we test:

• Python 3.6: cimg/python:3.6.9
• Python 3.7: cimg/python:3.7.10
• Python 3.8: cimg/python:3.8.9
• Python 3.9: cimg/python:3.9.5
• Python 3.10: cimg/python:3.10.1

Test workflow definitions

As per DRY principles, the test workflows are defined once as a template, for cimg/python:3.8.9, and reused for each of the other Docker images (defined at the bottom of the config.yml file).

Testing Environment Setup

The working directory for testing in each image is ~/repo.

The first action is that apt-get is updated, and upgraded, to ensure the base package manager is up-to-date, as apt-get is then used to install third-party tools native to the system:

• csh
• mummer
• ncbi-blast+
• fastani

The unsupported, deprecated, legacy BLAST tool is not available through apt-get or any of the other package managers, so this is installed as binaries downloaded from the NCBI FTP site.

curl -o legacy_blast.tar.gz ftp://ftp.ncbi.nlm.nih.gov/blast/executables/legacy.NOTSUPPORTED/2.2.26/blast-2.2.26-x64-linux.tar.gz
tar -zxf legacy_blast.tar.gz
echo 'export PATH=$PWD/blast-2.2.26/bin:$PATH' >> $BASH_ENV
source $BASH_ENV

fastANI is also unavailable through these routes.

wget https://github.com/ParBLiSS/FastANI/releases/download/v1.33/fastANI-Linux64-v1.33.zip
unzip fastANI-Linux64-v1.33.zip
echo 'export PATH=$PWD:$PATH' >> $BASH_ENV
source $BASH_ENV
fastANI -h

Python/pyani dependencies are installed using pip. The setuptools and virtualenv packages are updated prior to dependency installation:

pip install --upgrade pip setuptools virtualenv
pip install -r requirements.txt
pip install -r requirements-dev.txt
pip install -r requirements-pip.txt
pip install -r requirements-pyqt-pip.txt

pyani is then installed from the repository using the --editable option:

pip install -e .

Test Strategy

The pytest test suite includes integration tests (test_subcmd_*.py) and unit tests (the rest). Some tests take an appreciable time to execute on the CircleCI images and can cause the run to exceed the allowed time without a response. As a consequence, we subdivide the tests into slow and not slow tests, that are run separately. The slow tests are mostly tests of the legacy scripts, and these are expected to be deprecated at some future date.

The tests are run as two distinct jobs:

python -m pytest -v -m "not slow" --cov=pyani --cov-report xml:.coverage_fast.xml
python -m pytest -v -m "slow" --cov=pyani --cov-report xml:.coverage_slow.xml

The tests write two difference coverage output files: .coverage_fast.xml and .coverage_slow.xml.

Test Artifacts

Two kinds of test artifacts are produced: test output, and coverage results. All artifacts are exposed via the CircleCI web interface.

The test output is written to tests/test_output, and is the same output as would be written with local testing, using pytest -v at the command-line.

The coverage_results are separate XML files for the slow and not slow tests. These are uploaded to CodeCov, where they are merged (see this page for details).

Dependency Caching

We use CircleCI's ability to cache dependendencies. Each unique combination of Python version and requirements file is used to generate a cache of the /home/circleci/.pyenv directory, minimising redownload of Python dependencies. The cache key is defined as:

pyani-dependencies-pip-{{ .Environment.CIRCLE_JOB }}-{{ .Branch }}-{{ checksum "requirements.txt" }}-{{ checksum "requirements-dev.txt" }}-{{ checksum "requirements-pip.txt" }}-{{ checksum "requirements-thirdparty.txt" }}-{{ checksum "requirements-fastani.txt" }}-{{ checksum "requirements-pyqt-conda.txt" }}-{{ checksum "requirements-pyqt-pip.txt" }}

which combines the CIRCLE_JOB environment variable (i.e. the job name) with the branch being tested, and checksums for each of the requirements files.