A collection of small scripts, adhering to the Unix philosophy, used to release The Foreman. They automate parts of the branch and release processes. It is also used for the Katello branch and release processes.
The most important environment variables are PROJECT and VERSION.
Procedures are kept in the procedures directory.
Both the branch and release procedures defines two roles: an owner and an engineer. Someone can be both owner and engineer at the same time, and it is preferable for a fast process. Having both in (very) different timezones can lengthen the process. Sometimes it's just not possible and multiple people perform the procedures. Below is a listing of access used.
- Belong to the Foreman GitHub release-managers team
- Belong to the releases group on Discourse
- Matrix: permission to change the topic in #theforeman on matrix.org. See Matrix moderation on how channel admins can grant this.
- Redmine access: be added to the Developers group
- Transifex account (unclear on the exact permissions to extract translations)
- Belong to the Foreman GitHub release-engineering team
- Foreman infrastructure access including secrets
- Jenkins access
- web01 stagingyum access
- theforeman Fedora group member which can be viewed through Fedora Account System
- Write permission to Katello/katello
- Push access to the Katello gem
- Belong to the Foreman GitHub katello-release-owners team
- Belong to the
releasesgroup on Discourse - Redmine access: be added to the Developers group
These are the same as Foreman's release engineer.
Use procedure_branch to display the steps. Modify PROJECT and VERSION as needed.
PROJECT=foreman VERSION=3.7 ./procedure_branchIt will have created releases/${PROJECT}/${VERSION}/settings, which should be submitted as a pull request.
A more advanced invocation is to specify the parameters directly and copy the output.
This uses wl-copy for Wayland; xclip achieves the same for Xorg.
PROJECT=foreman VERSION=3.7 ./procedure_branch 2023-05-23 TheOwner TheEngineer | wl-copy
Now post this to Discourse Development/Releases as $PROJECT $VERSION branching process.
Follow the process as instructed.
First make sure you have fetched, pulled in the latest commits and rebased your fork. Secondly ensure releases/${PROJECT}/${VERSION}/settings contains the correct FULLVERSION. Modify as needed and submit as a pull request.
Then generate the procedure, similar to the branching process:
PROJECT=foreman VERSION=3.7 ./procedure_releaseOr the complete version:
PROJECT=foreman VERSION=3.7 ./procedure_release 2023-05-23 TheOwner TheEngineer | wl-copy
Now post this to Discourse Development/Releases as $PROJECT $FULLVERSION release process.
Follow the process as instructed.
Python and python-jenkins are needed. On Fedora this can be installed:
dnf install python3-jenkinsForeman requires GPG signed git tags. Configure git with your personal gpg key id as your signing key:
git config --global user.signingKey <YOUR_PUB_KEY_HASH>
For storing secrets, gopass is used. On Fedora it can be installed:
dnf install gopassThen make sure you have a GPG key on your system and initialize gopass with your key:
gopass init <YOUR-PUB-KEY-HASH>
For running jobs in Jenkins, make sure you have access and add your Jenkins password or API token:
gopass edit theforeman/jenkins-token --create
Copr is where all packages are built and the stage repositories are generated from for releases. API access is needed in order to perform release activities. To ensure you have Copr access setup:
- Go to https://copr.fedorainfracloud.org/api/
- Login if you are not already
- Follow the directions
For commands on the Foreman infrastructure, add your personal sudo password:
gopass edit theforeman/unix --create
The releases store from the shared secret storage is also needed:
gopass clone secrets.theforeman.org:/srv/secretsgit/theforeman-release.git theforeman/releases
When a GPG key has already been generated, it can be imported from the backups:
import_gpg_privateMake sure VERSION is correct in settings and FULLVERSION in releases/$PROJECT/$VERSION/settings. This assumes a GPG key is already present.
./tag_project
./release_tarballs
./download_tarballs
./inspect_tarballs
./sign_tarballs
./bump_deb_packaging
./bump_rpm_packaging
./release_packages
# These steps can happen during the build after RPMs have been built but DEBs are still running
./generate_stage_repository
./sign_stage_rpms
./upload_stage_rpmsWhen handling non-Foreman releases (currently supported: Katello and Client), set PROJECT to the lowercase name of the project and VERSION to the version of the project (if it differs from the Foreman one).
PROJECT=client ./generate_stage_repository
PROJECT=katello VERSION=3.13 ./generate_stage_repositoryWhen starting a new release, the following scripts can be used to generate a new key:
generate_gpg
export_gpg_private
export_gpg_public
sign_gpg
upload_gpgOur Debian repository doesn't rotate keys based on releases, but on a time basis.
To generate a new key:
export PROJECT=foreman-debian
export VERSION="$(date '+%Y')"
generate_gpg
export_gpg_private
export_gpg_public
sign_gpg
upload_gpgSettings can be customized in settings.local. The following settings are supported:
GIT_DIR=$HOME/dev # Projects are cloned here
GIT_REMOTE=upstream # Git remote for cloned projects
PACKAGING_PR=true # Create a PR in bump_{deb,rpm}_packagingThe build_stage_repository script generates a stage repository locally that can then be uploaded to the staging repository server or used locally for testing. This is done for RPMs and SRPMs. The workflow for each is:
For releases (executed by release owner):
1. Uses reposync to copy all RPMs from production (yum.theforeman.org) to a local repository
2. Run repodiff comparing the local copy of production and Copr for the release
3. Download the new packages from Copr
4. Filter all packages through comps file, removing anything not in the comps file
5. Runs createrepo
6. Generates a new module metadata file based upon the local repository
7. Generate list of unsigned RPMs
8. Sign the unsigned RPMs
9. Update the repository metadata
For nightly (executed by Jenkins):
1. Copy all RPMs from Copr for the given repository to local
2. Filter the downloaded RPMs through comps file, removing anything not in the comps file
3. Runs createrepo
4. Generates a new module metadata file based upon the local repository
The local repository can then be uploaded to the server backing stagingyum.theforeman.org using rsync. Where any RPM found locally already in staging is kept, and any RPM found in staging not in the local repository is removed.
The release workflow is execute using the following actions:
generate_stage_repository
sign_stage_repository
upload_stage_repository