From 4cc3085be2932e99d1c28eaa8e5d10049304d9c7 Mon Sep 17 00:00:00 2001 From: Rob E Date: Tue, 4 Aug 2026 16:30:52 +1000 Subject: [PATCH 1/5] Adding notes no release process --- docs/README.md | 99 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 99 insertions(+) create mode 100644 docs/README.md diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..bf6fe06 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,99 @@ +## Image Release Process + +**Note: Unless a semantic version build tag is set on the commit being triggered, TeamCity will generate a pre-release build of the images. This should not go out to Production** + +## 👷 Build +When a change is made to a file in this repository, TeamCity triggers the [full build chain](https://build.octopushq.com/buildConfiguration/OctopusDeploy_WorkerTools_ChainBuildTestAndRelease). +This results in a build of all the different distro's dockerfiles, as well as seperate architecture for non-windows os. + +As part of this build process, individual images are pushed to the internal artifactory image repository. + +## 🐙 Deploy +As the end of a successful build, a release is created in the [Worker Tools vNext](https://deploy.octopus.app/app#/Spaces-1163/projects/worker-tools-vnext/deployments?groupBy=Channel") Octopus Deploy Project. + +This project takes the individually built images from Artifactory, and pushes them to a target repository as well as a multi-arch image to that same registry. As we create multiple architecture images for `Ubuntu 24.04` (amd64 and arm64), an additional `ubuntu.24.04` multi-arch image is created and pushed. + + +For the `Staging` environment the target is the same Artifactory registry where they are pushed during the build chain so then individual image push essentially becomes a no-op. The multi-arch images however will still be pushed. +For `Production` the target repository is the [public DockerHub registry](https://hub.docker.com/r/octopusdeploy/worker-tools). + +## Flow Diagram + +```mermaid +flowchart TD + subgraph repo[🧑‍💻 This Git Repository] + direction LR + +RepoDirs@{ shape: docs, label: "Per Platform Docker Config + +* _ubuntu.22.04_ +* _ubuntu.24.04_ +* _windows.ltsc2022_ +* _windows.ltsc2025_"} + end + + subgraph ci[👷 Team City] + Chain[Chain: Build Test & Release] + ReleaseAll[Release:All] + Builds@{ shape: docs, label: "Per Platform Build Config + +* _ubuntu 22.04_ +* _ubuntu 22.04 x64_ +* _Windows 2022_ +* _Windows 2025_"} + Chain --> ReleaseAll + ReleaseAll --> Builds + + + + click Chain "https://build.octopushq.com/project/OctopusDeploy_WorkerTools?mode=builds#all-projects" "TeamCity" + end + + subgraph Artifactory[📦 Artifactory] + ArtifactoryWorkerTools[docker.packages.octopushq.com/octopusdeploy/workertools] + end + + subgraph DockerHub[📦 Docker Hub] + DockerHubWorkerTools[docker.io/octopusdeploy/worker-tools] + click DockerHubWorkerTools "https://hub.docker.com/r/octopusdeploy/worker-tools" + end + + subgraph Octopus[🐙 Octopus Deploy] + subgraph Project[Worker Tools vNext] + + Staging + Prod + Staging-->Prod + end + click Prod "https://deploy.octopus.app/app#/Spaces-1163/projects/worker-tools-vnext/deployments?groupBy=Channel" + click Staging "https://deploy.octopus.app/app#/Spaces-1163/projects/worker-tools-vnext/deployments?groupBy=Channel" + end + + Builds --> ArtifactoryWorkerTools + +Artifactory -.-> Staging +Artifactory -.-> Prod +Staging --> |Push MultiArch Image|ArtifactoryWorkerTools +Prod --> |Push OS Images|DockerHubWorkerTools +Prod --> |Push MultiArch Image|DockerHubWorkerTools +Prod --> |Push Latest Tag|DockerHubWorkerTools +repo moving2@==> |CI trigger|ci + +ReleaseAll moving1@==> |On Success|Staging + +moving1@{ animate: true } +moving2@{ animate: true } + +``` + +## Consumption in Octopus Deploy +When presented with the option to run an `Execution Container Image`, users can select this pre-built worker-tool image. + +image + +The list of images presented is dynamically retrieved from DockerHub and so, once the image has been pushed to `Production`, it potentially becomes the default image that customer will use when configuring a new step. + +Unfortunately if the image is not yet cached on the Dynamic Worker VM Image, cloud customers may find that the image pull process takes an unreasonable amount of time. For this reason, it is reccomended that pushing to `Production` is only done once the new image has been sucessfully cached in the available Dynamic Worker VM. + +## Dynamic Worker Images +TODO: Add To This From 6b0bf88e1df81b45e1f55274a79de75879877ca1 Mon Sep 17 00:00:00 2001 From: Rob E Date: Tue, 11 Aug 2026 09:14:28 +1000 Subject: [PATCH 2/5] Revise README for clarity and additional information Updated README to enhance clarity and add new sections regarding artifact storage and dynamic worker images. --- docs/README.md | 96 +++++++++----------------------------------------- 1 file changed, 17 insertions(+), 79 deletions(-) diff --git a/docs/README.md b/docs/README.md index bf6fe06..e2ad6ad 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,6 +1,8 @@ ## Image Release Process -**Note: Unless a semantic version build tag is set on the commit being triggered, TeamCity will generate a pre-release build of the images. This should not go out to Production** +**⚠️ Note: Unless a semantic version build tag is set on the commit being triggered, TeamCity will generate a pre-release build of the images. This should not go out to Production** + +`GitHub` Dockerfile Per Platform --> `TeamCity` [Chain:ReleaseAll](https://build.octopushq.com/project/OctopusDeploy_WorkerTools?mode=builds#all-projects) --> `Octopus (Dev)` Push to Artifactory --> `Octopus (Prod)` Push to DockerHub ## 👷 Build When a change is made to a file in this repository, TeamCity triggers the [full build chain](https://build.octopushq.com/buildConfiguration/OctopusDeploy_WorkerTools_ChainBuildTestAndRelease). @@ -8,83 +10,18 @@ This results in a build of all the different distro's dockerfiles, as well as se As part of this build process, individual images are pushed to the internal artifactory image repository. -## 🐙 Deploy -As the end of a successful build, a release is created in the [Worker Tools vNext](https://deploy.octopus.app/app#/Spaces-1163/projects/worker-tools-vnext/deployments?groupBy=Channel") Octopus Deploy Project. - -This project takes the individually built images from Artifactory, and pushes them to a target repository as well as a multi-arch image to that same registry. As we create multiple architecture images for `Ubuntu 24.04` (amd64 and arm64), an additional `ubuntu.24.04` multi-arch image is created and pushed. - - -For the `Staging` environment the target is the same Artifactory registry where they are pushed during the build chain so then individual image push essentially becomes a no-op. The multi-arch images however will still be pushed. -For `Production` the target repository is the [public DockerHub registry](https://hub.docker.com/r/octopusdeploy/worker-tools). - -## Flow Diagram - -```mermaid -flowchart TD - subgraph repo[🧑‍💻 This Git Repository] - direction LR - -RepoDirs@{ shape: docs, label: "Per Platform Docker Config - -* _ubuntu.22.04_ -* _ubuntu.24.04_ -* _windows.ltsc2022_ -* _windows.ltsc2025_"} - end - - subgraph ci[👷 Team City] - Chain[Chain: Build Test & Release] - ReleaseAll[Release:All] - Builds@{ shape: docs, label: "Per Platform Build Config - -* _ubuntu 22.04_ -* _ubuntu 22.04 x64_ -* _Windows 2022_ -* _Windows 2025_"} - Chain --> ReleaseAll - ReleaseAll --> Builds - +## 📦 Artifact Storage +* **Artifactory**: (pre-prod) _docker.packages.octopushq.com/octopusdeploy/workertools_ +* **DockerHub** (prod) [docker.io/octopusdeploy/worker-tools](https://hub.docker.com/r/octopusdeploy/worker-tools) - - click Chain "https://build.octopushq.com/project/OctopusDeploy_WorkerTools?mode=builds#all-projects" "TeamCity" - end - - subgraph Artifactory[📦 Artifactory] - ArtifactoryWorkerTools[docker.packages.octopushq.com/octopusdeploy/workertools] - end - - subgraph DockerHub[📦 Docker Hub] - DockerHubWorkerTools[docker.io/octopusdeploy/worker-tools] - click DockerHubWorkerTools "https://hub.docker.com/r/octopusdeploy/worker-tools" - end - - subgraph Octopus[🐙 Octopus Deploy] - subgraph Project[Worker Tools vNext] - - Staging - Prod - Staging-->Prod - end - click Prod "https://deploy.octopus.app/app#/Spaces-1163/projects/worker-tools-vnext/deployments?groupBy=Channel" - click Staging "https://deploy.octopus.app/app#/Spaces-1163/projects/worker-tools-vnext/deployments?groupBy=Channel" - end - - Builds --> ArtifactoryWorkerTools - -Artifactory -.-> Staging -Artifactory -.-> Prod -Staging --> |Push MultiArch Image|ArtifactoryWorkerTools -Prod --> |Push OS Images|DockerHubWorkerTools -Prod --> |Push MultiArch Image|DockerHubWorkerTools -Prod --> |Push Latest Tag|DockerHubWorkerTools -repo moving2@==> |CI trigger|ci - -ReleaseAll moving1@==> |On Success|Staging - -moving1@{ animate: true } -moving2@{ animate: true } - -``` +## 🐙 Deploy +The deployment process in [Worker Tools vNext](https://deploy.octopus.app/app#/Spaces-1163/projects/worker-tools-vnext/deployments?groupBy=Channel) performs some Image manipulation/pushing. +1. Pull each Image from Artifactory (where TC Build Pushes each) +2. Push each Image to target registry (results in a no-op for pre-prod which uses the same Artifactory registry it just pushed from +3. Create multi-arch `:` taged image across all distros and push to target registry +4. Create multi-arch `:ubuntu-` image and push to target registry (Windows does not support ARM so no need) +5. +**⚠️ Note: Until the Cloud infrastructure has been updated to use a Dynamic Worker VM that contains a cached version of the newly created image then it should not be pushed to production. See Below for more details.** ## Consumption in Octopus Deploy When presented with the option to run an `Execution Container Image`, users can select this pre-built worker-tool image. @@ -95,5 +32,6 @@ The list of images presented is dynamically retrieved from DockerHub and so, onc Unfortunately if the image is not yet cached on the Dynamic Worker VM Image, cloud customers may find that the image pull process takes an unreasonable amount of time. For this reason, it is reccomended that pushing to `Production` is only done once the new image has been sucessfully cached in the available Dynamic Worker VM. -## Dynamic Worker Images -TODO: Add To This +## ☁️ Octopus Cloud Dynamic Worker Images +Not until the +[Cloud Platform deploy space](https://deploy.octopus.app/app#/Spaces-142/projects?searchValue=Dynamic+Worker+Images&page=1&pageSize=50&hideInactiveProjects=false) contains a project per VM. From 9026ccd29878fb197f28b8de0a270579592f41a7 Mon Sep 17 00:00:00 2001 From: Rob E Date: Thu, 13 Aug 2026 11:59:17 +1000 Subject: [PATCH 3/5] Update README.md --- docs/README.md | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/docs/README.md b/docs/README.md index e2ad6ad..44d737a 100644 --- a/docs/README.md +++ b/docs/README.md @@ -2,7 +2,11 @@ **⚠️ Note: Unless a semantic version build tag is set on the commit being triggered, TeamCity will generate a pre-release build of the images. This should not go out to Production** -`GitHub` Dockerfile Per Platform --> `TeamCity` [Chain:ReleaseAll](https://build.octopushq.com/project/OctopusDeploy_WorkerTools?mode=builds#all-projects) --> `Octopus (Dev)` Push to Artifactory --> `Octopus (Prod)` Push to DockerHub +1. `GitHub` Dockerfile Per Platform +2. `TeamCity` [Chain:ReleaseAll](https://build.octopushq.com/project/OctopusDeploy_WorkerTools?mode=builds#all-projects) +3. `Octopus (Dev)` Push to Artifactory +4. Wait For Worker Tools VM to push to Production (see below) +5. `Octopus (Prod)` Push to DockerHub ## 👷 Build When a change is made to a file in this repository, TeamCity triggers the [full build chain](https://build.octopushq.com/buildConfiguration/OctopusDeploy_WorkerTools_ChainBuildTestAndRelease). @@ -35,3 +39,16 @@ Unfortunately if the image is not yet cached on the Dynamic Worker VM Image, clo ## ☁️ Octopus Cloud Dynamic Worker Images Not until the [Cloud Platform deploy space](https://deploy.octopus.app/app#/Spaces-142/projects?searchValue=Dynamic+Worker+Images&page=1&pageSize=50&hideInactiveProjects=false) contains a project per VM. + +Go to the Image deployment (e.g [x](https://build.octopushq.com/buildConfiguration/CloudPlatform_DynamicWorkers_DynamicWorkerVmImages_Ubuntu2204/23691299?showLog=23691299_2731_39.67.645.2501.2731&logView=flowAware)), find the latest build and look for the line entry describing the cached versions, looking for the header `Cache Docker Images`. At the bottom of this you will see log entry +``` +azure-arm.ubuntu: Selecting the following images due to maximum image count of 3 : +azure-arm.ubuntu: - "6.6.5-ubuntu.22.04" +azure-arm.ubuntu: - "6.6.4-ubuntu.22.04" +azure-arm.ubuntu: - "6.6.3-ubuntu.22.04" +``` + +For each of the relevant distros, check the earliest build that had the new docker image cached. + +Once _all_ of these builds have been deployed to Production from the [Dynamic Worker Images Project](https://deploy.octopus.app/app#/Spaces-142/projects?includeProjectGroupId=ProjectGroups-1841&searchValue=Dynamic+Worker+Images&page=1&pageSize=50&hideInactiveProjects=false) Group, then the docker images can be released to production. + From 1e2204fdd21905a8800db4213ea3b1758ffff7ab Mon Sep 17 00:00:00 2001 From: Rob E Date: Thu, 13 Aug 2026 12:06:42 +1000 Subject: [PATCH 4/5] Update README.md --- docs/README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/README.md b/docs/README.md index 44d737a..5c057b1 100644 --- a/docs/README.md +++ b/docs/README.md @@ -5,7 +5,7 @@ 1. `GitHub` Dockerfile Per Platform 2. `TeamCity` [Chain:ReleaseAll](https://build.octopushq.com/project/OctopusDeploy_WorkerTools?mode=builds#all-projects) 3. `Octopus (Dev)` Push to Artifactory -4. Wait For Worker Tools VM to push to Production (see below) +4. Wait For Worker Tools VM to push to Production Tenants (see below) 5. `Octopus (Prod)` Push to DockerHub ## 👷 Build @@ -50,5 +50,7 @@ azure-arm.ubuntu: - "6.6.3-ubuntu.22.04" For each of the relevant distros, check the earliest build that had the new docker image cached. +Each Octopus Project has a `Cautious Promoter` step which runs the [Cautious Promoter](https://github.com/OctopusDeploy/CautiousPromoter) tool to determin when to push to each Production Tenant. + Once _all_ of these builds have been deployed to Production from the [Dynamic Worker Images Project](https://deploy.octopus.app/app#/Spaces-142/projects?includeProjectGroupId=ProjectGroups-1841&searchValue=Dynamic+Worker+Images&page=1&pageSize=50&hideInactiveProjects=false) Group, then the docker images can be released to production. From cc14ce35728b8fccf42288f5c0ba2865ad7340aa Mon Sep 17 00:00:00 2001 From: Rob E Date: Thu, 3 Sep 2026 15:51:55 +1000 Subject: [PATCH 5/5] Move execution container images --- docs/README.md | 1 + docs/execution-container-images.md | 52 ++++++++++++++++++++++++++++++ 2 files changed, 53 insertions(+) create mode 100644 docs/execution-container-images.md diff --git a/docs/README.md b/docs/README.md index 5c057b1..531a900 100644 --- a/docs/README.md +++ b/docs/README.md @@ -54,3 +54,4 @@ Each Octopus Project has a `Cautious Promoter` step which runs the [Cautious Pro Once _all_ of these builds have been deployed to Production from the [Dynamic Worker Images Project](https://deploy.octopus.app/app#/Spaces-142/projects?includeProjectGroupId=ProjectGroups-1841&searchValue=Dynamic+Worker+Images&page=1&pageSize=50&hideInactiveProjects=false) Group, then the docker images can be released to production. +See [here](./execution-container-images.md) for further information about how execution container images work for Hosted Octopus. diff --git a/docs/execution-container-images.md b/docs/execution-container-images.md new file mode 100644 index 0000000..bb71a97 --- /dev/null +++ b/docs/execution-container-images.md @@ -0,0 +1,52 @@ +# Execution Container Images in Hosted Octopus + +[Execution Containers](https://octopus.com/docs/projects/steps/execution-containers-for-workers) run Docker images like [Worker Tools](https://github.com/OctopusDeploy/WorkerTools). These container images provide a common tool set to support the execution of steps, and are pre-cached on [Dynamic Worker VMs](https://octopus.com/docs/infrastructure/workers/dynamic-worker-pools). This documentation describes some architectural designs for these images to improve user experience. + +## How Execution Container Images are Released + +Previously, when a new version of Worker Tools was created, it was released directly to DockerHub. This became immediately available to users but Dynamic Workers didn't have this version cached yet. Whenever a new Worker Tools image is published, there's a 1-2 week roll-out period where the latest image isn't cached on Dynamic Workers. This caused slow deployments for users updating their Worker Tools due to workers downloading the image. Additionally pushing images directly to DockerHub was also against the principles of Sensible Defaults. + +To improve customer experience, changes to the Worker Tools release process have been made. We use [JFrog Artifactory](https://packages.octopushq.com/ui/packages) as the staging environment for releasing new images. A new image is first pushed Artifactory. The Dynamic Workers VM Image pipeline will pull from Artifactory to cache recent images and produce a VM image. Once this VM image is released, new Dynamic Workers will be created with the latest version of Worker Tools cached. The process of selecting versions of images to cache is described in the [next section](#which-versions-to-cache). + +Since it can take weeks for the Dynamic Workers with the new image cached to reach all production reefs, we created an [Execution Container Promoter](https://github.com/OctopusDeploy/ExecutionContainersPromoter). The Promoter is a Runbook that periodically checks whether an image version has reached all production reefs, and publishes it to DockerHub if it has. The detailed working mechanism of the Promoter can be found in [repo](https://github.com/OctopusDeploy/ExecutionContainersPromoter). + +Finally, once an image is promoted to DockerHub, we can say for sure that the version is readily cached for all production Dynamic Workers. The Octopus Server UI checks this registry to guide customers to use the latest version. + +```mermaid +sequenceDiagram + participant wtb as Worker Tools build + participant a as Artifactory (Staging) + participant dwb as Dynamic Worker VM Image build + participant p as Promoter + participant d as DockerHub (Production) + participant u as UI + + wtb ->> a: Push to + dwb ->> a: Pull from + loop + p -->> dwb: check if cached in prod + end + p ->> d: Push to + u -->> d: Get and suggest latest tag + +``` + +## Which Versions to Cache + +When the Dynamic Worker VM images are built, the most recent versions of Worker Tools images will be pulled from Artifactory and cached. + +Previously, only one version of the images was cached. This caused long downloading time for some customers, especially those using Windows. We encouraged users to configure an absolute version number for Worker Tools. When this version was no longer the latest, slow deployments would occur. + +To reduce the chance of introducing unexpected slow deployments, we now cache multiple versions of the Worker Tools. The [algorithm](https://github.com/OctopusDeploy/DynamicWorkerVmImages/blob/master/Win2019/scripts/cache-docker-images.ps1) depicted in the following diagram is used to select versions to cache, which balances the disk space on Dynamic Worker VMs and the popularity of the Worker Tools images being used. + +```mermaid +flowchart TD + A(All versions published on Artifactory) --> B(Remove any ad-hoc suppressed versions) + B --> C(Get latest N versions
N = 5 for Ubuntu
N = 3 for Windows) + B --> D(Get latest N major versions
N = 3 for Ubuntu
N = 2 for Windows) + C --> E(Combine) + D --> E + E --> F(Add any ad-hoc versions to cache) + F --> G(Take at most N
N = 5 for Ubuntu
N = 2 for Windows) + G --> H(Versions to cache) +```