Skip to content

Latest commit

 

History

History
350 lines (233 loc) · 19.4 KB

Develop.md

File metadata and controls

350 lines (233 loc) · 19.4 KB
Raw

Devops notes for nicobot

Build Status on 'master' branch PyPi
Build and publish to Docker Hub
Docker debian Docker signal-debian Docker alpine

Basic development

Install Python dependencies (for both building and running) and generate nicobot/version.py with :

pip3 install -c constraints.txt -r requirements-build.txt -r requirements-runtime.txt
python3 setup.py build

To run unit tests :

python3 -m unittest discover -v -s tests

To run directly from source (without packaging) :

python3 -m nicobot.askbot [options...]

To build locally (more at pypi.org) :

rm -rf ./dist ; python3 setup.py build sdist bdist_wheel

PyPi upload

To upload to test.pypi.org :

python3 -m twine upload --repository testpypi dist/*

To install the test package from test.pypi.org and check that it works :

# First create a virtual environment not to mess with the host system
python3 -m venv venv/pypi_test && source venv/pypi_test/bin/activate

# Then install dependencies using the regular pypi repo
pip3 install -c constraints.txt -r requirements-runtime.txt

# Finally install this package from the test repo
pip3 install -i https://test.pypi.org/simple/ --no-deps nicobot

# Do some test
python -m nicobot.askbot -V
...

# Exit the virtual environment
deactivate

To upload to PROD pypi.org :

python3 -m twine upload dist/*

Both above twine upload commands will ask for a username and a password. To prevent this, you could set variables :

# Defines username and password (or '__token__' and API key)
export TWINE_USERNAME=__token__
# Example reading the token from a local 'passwordstore'
export TWINE_PASSWORD=`pass pypi/test.pypi.org/api_token`

Or store them in ~/.pypirc (see doc) :

[pypi]
username = __token__
password = <PyPI token>

[testpypi]
username = __token__
password = <TestPyPI token>

Or even use CLI options -u and -p, or certificates... See python3 -m twine upload --help for details.

Automation for PyPi

The above instructions allow to build manually but otherwise it is automatically tested, built and uploaded to pypi.org using Travis CI on each push to GitHub (see .travis.yml).

Docker build

There are several Dockerfiles, each made for specific use cases (see README.md). They all have multiple stages.

debian.Dockerfile is quite straight. It builds using pip in one stage and copies the resulting wheels into the final one.

signal-debian.Dockerfile is more complex because it needs to address :

  • including both Python and Java while keeping the image size small
  • compiling native dependencies (both for signal-cli and qr)
  • circumventing a number of bugs in multiarch building

alpine.Dockerfile produces smaller images but may not be as much portable than debian ones and misses Signal support for now.

Note that the signal-cli backend needs a Java runtime environment, and also rust dependencies to support Signal's group V2. This approximately doubles the size of the images and almost ruins the advantage of alpine over debian...

Those images are limited on each OS (debian+glibc / alpine+musl) to CPU architectures which :

  1. have base images (python, openjdk, rust)
  2. have Python dependencies have wheels or are able to build them
  3. can build libzkgroup (native dependencies for signal)
  4. have the required packages to build

At the time of writing, support is dropped for :

  • linux/s390x : lack of python:3 image (at least)
  • linux/riscv64 : lack of python:3 image (at least)
  • Signal backend on linux/arm* for Alpine variants : lack of JRE binaries

All images have all the bots inside (as they would otherwise only differ by one script from each other). The docker-entrypoint.sh script takes the name of the bot to invoke as its first argument, then its own options and finally the bot's arguments.

Sample build command (single architecture) :

docker build -t nicolabs/nicobot:debian -f debian.Dockerfile .

Sample buildx command (multi-arch) :

docker buildx build --platform linux/amd64,linux/arm64,linux/386,linux/arm/v7 -t nicolabs/nicobot:debian -f debian.Dockerfile .

Then run with the provided sample configuration :

docker run --rm -it -v "$(pwd)/tests:/etc/nicobot" nicolabs/nicobot:debian askbot -c /etc/nicobot/askbot-sample-conf/config.yml

Automation for Docker Hub

Github actions are currently used (see .github/workflows/dockerhub.yml to automatically build and push the images to Docker Hub so they are available whenever commits are pushed to the master branch :

  1. A Github Action is triggered on each push to the central repo
  2. Alpine images and Debian images are built in parallel to speed up things. Debian-signal is built after Debian. Caching is used for both. See .github/workflows/dockerhub.yml.
  3. Images are uploaded to Docker Hub

Tagging strategy

Since I could not find an easy way to generate exactly the tags I wanted, the setup.py script embeds a custom command to generate them from the git context (tag, commit) and the image variant :

Docker build process overview

This diagram is the view from the master branch on this repository. It emphasizes FROM and COPY relations between the images (base and stages).

nicobot docker images build process

Why no image is available for arch x ?

You may find the reason for a missing CPU architecture / combination within the open issues labelled with docker.

Docker image structure

Here are the main application files and directories inside the images :

📦 /
 ┣ 📂 etc/nicobot/ - - - - - - - - - - - -> Default configuration files
 ┃ ┣ 📜 config.yml
 ┃ ┣ 📜 i18n.en.yml
 ┃ ┗ 📜 i18n.fr.yml
 ┣ 📂 root/
 ┃ ┗ 📂 .local/
 ┃   ┣ 📂 bin/ - - - - - - - - - - - - - -> Executable commands
 ┃   ┃ ┣ 📜 askbot
 ┃   ┃ ┣ 📜 docker-entrypoint.sh
 ┃   ┃ ┣ 📜 transbot
 ┃   ┃ ┗ 📜 ...
 ┃   ┗ 📂 lib/pythonX.X/site-packages/ - -> Python packages (nicobot & dependencies)
 ┗ 📂 var/nicobot/  - - - - - - - - - - - -> Working directory & custom configuration files & data (contains secret stuff !)
   ┣ 📂 .omemo/ - - - - - - - - - - - - - -> OMEMO keys (XMPP)
   ┗ 📂 .signal-cli/  - - - - - - - - - - -> signal-cli configuration files

Deploying on AWS

This chapter describes a very simple way to deploy the bots on Amazon Web Services. There are many other methods and Cloud providers but you can build on this example to start implementing your specific case.

Here is the process :

  1. Get an AWS account
  2. Install the latest Docker Desktop or Docker Compose CLI with ECS support (make sure to start a new shell if you've just installed it)
  3. Configure the AWS credentials (with AWS_* environnement variables or ~/.aws/credentials)
  4. Create and switch your local docker to an 'ecs' context : docker context create ecs myecs && docker context use myecs
  5. Craft a docker-compose.yml file (see templates tests/transbot-jabber.docker-compose.yml and tests/transbot-signal.docker-compose.yml)
  6. Make sure you have the proper configuration files (only a config.yml is required in the given templates) and start the service : docker compose up

If you follow the given templates :

  • this will deploy nicobot on AWS' Fargate
  • the given config.yml file will be injected as a secret
  • it will use the writable layer of the container to download translation files and generate temporary files like OMEMO keys
  • if you use the Signal backend it should print the QRCode to scan at startup ; you should also find the URI to manually generate it in the logs on CloudWatch console
  • once done, docker compose down will stop the bot by clearing everything from AWS

If you want to customize the image, you have the option to upload it to a private registry on AWS before deploying your stack :

  1. First make a copy of tests/transbot-sample-conf/sample.env and set the variables inside according to your needs. Let's say you've put it at tests/transbot-sample-conf/aws.env. Image-related variables should look like : NICOBOT_IMAGE=123456789012.dkr.ecr.eu-west-1.amazonaws.com/nicobot and NICOBOT_BASE_IMAGE=123456789012.dkr.ecr.eu-west-1.amazonaws.com/nicobot:dev-signal-debian (see ECR docs
  2. Make sure to authenticate against your private registry - tip : use Amazon ECR Docker Credential Helper for a seamless integration with the docker command line
  3. Build the image with docker-compose (docker compose on AWS doesn't support build nor push) : cd tests/transbot-sample-conf && docker-compose build
  4. Push the image to your private AWS ECR[^1][^2] : docker-compose --env-file aws.env push
  5. Finally, deploy as before : docker context use myecs && docker compose --env-file aws.env up

As this method relies on a standard docker-compose file, it is very straightforward and also works on a developer workstation (simply replace docker compose with docker-compose). However it cannot go beyond the supported mappings with CloudFormation templates (the native AWS deployment descriptor) and AWS's choice of services (Fargate, EFS, ...). In addition, as seen above, you currently have to use different commands (docker-compose / docker compose) to build & push or deploy.

Versioning

The --version command-line option that displays the bots' version relies on setuptools_scm, which extracts it from the underlying git metadata. This is convenient because the developer does not have to manually update the version (or forget to do it), however it either requires the version to be fixed inside a Python module or the .git directory to be present.

There were several options among which the following one has been retained :

  1. Running setup.py creates / updates the version inside the version.py file
  2. The scripts then load this module at runtime

Since the version.py file is not saved into the project, setup.py build must be run before the version can be queried. In exchange :

  • it does not require setuptools nor git at runtime
  • it frees us from having the .git directory around at runtime ; this is especially useful to make the docker images smaller

Tip : python3 setup.py --version will print the guessed version.

Building signal-cli

The signal backend (actually signal-cli) requires a Java runtime, which approximately doubles the image size. This led to build separate images (same repo but different tags), to allow using smaller images when only the XMPP backend is needed.

Resources

AWS

IBM Cloud

Signal

Jabber

Python libraries

Dockerfile

JRE + Python in Docker

Multiarch & native dependencies

Python build & Python in Docker

Rust