v5.4.0

v5.4.0 (2021-06-30)

Added

• Docker Buildx #871
• Docs on which user SSH access applies to #863 (@Temikus)

Changed

• Update Buildkite Agent to version 3.30.0 #868 (@esalter)
• The HttpPutResponseHopLimit from 1 to 2 #858

Fixed

• The default cost allocation tag value #859

Upgrading

Perform a CloudFormation stack update with the following URL:
https://s3.amazonaws.com/buildkite-aws-stack/v5.4.0/aws-stack.yml

If you want to launch a new stack, you can use this link (make sure not to use your production AWS account, create a new one for CI):

Documentation

See the Readme for this release.

v5.3.2

v5.3.2 (2021-06-11)

Fixed

• s3secrets-helper for Windows #846 (DuBistKomisch)
• Docker systemd configuration to match the installed Docker version #849 (cmanou)
• Excessive instance scaling while waiting for instances to boot

Changed

• Create S3 secrets bucket only when needed #844 (vgrigoruk)

Upgrading

Perform a CloudFormation stack update with the following URL:
https://s3.amazonaws.com/buildkite-aws-stack/v5.3.2/aws-stack.yml

If you want to launch a new stack, you can use this link (make sure not to use your production AWS account, create a new one for CI):

Documentation

See the Readme for this release.

v5.3.1

v5.3.1 (2021-05-05)

Fixed

• Allow dashes and multiple forward slashes (/) in BuildkiteAgentTokenParameterStorePath #835 #837 (nitrocode)

Upgrading

Perform a CloudFormation stack update with the following URL:
https://s3.amazonaws.com/buildkite-aws-stack/v5.3.1/aws-stack.yml

If you want to launch a new stack, you can use this link (make sure not to use your production AWS account, create a new one for CI):

Documentation

See the Readme for this release.

v5.3.0

v5.3.0 (2021-04-28)

Added

• Support IAM Permissions Boundaries #767 #805 (nitrocode)
• Session manager plugin #818 (nitrocode)

Changed

• Replace awslogs with the cloudwatch-agent #811 (yob)
• Avoid scaling down too aggressively when there are pending jobs in certain conditions #823 (yob)
• Bump docker from 19.03.x to 20.10.x #826 (yob)
• Bump docker-compose on all operating systems to 1.28.x #825 (yob)
• Bump agent from 3.27.0 to 3.29.0 #827 (yob)
• Bump lifecycled from 3.0.2 to 3.2.0 #824 (yob)
• Bump git on windows from 2.22.0 to 2.31.0 #819 (yob)
• Bump ECR plugin to v2.3.0 #816 (chloeruka)
• Documentation improvements #815 #810 (acaire)

Removed

• Remove unnecessary IAM roles for SNS and SQS #829 (chloeruka)

Upgrading

Perform a CloudFormation stack update with the following URL:
https://s3.amazonaws.com/buildkite-aws-stack/v5.3.0/aws-stack.yml

If you want to launch a new stack, you can use this link (make sure not to use your production AWS account, create a new one for CI):

Documentation

See the Readme for this release.

Elastic CI Stack for AWS v5.2.0

Added

• buildkite-agent v3.27.0 #794 (pda)
• agent names use client-side %spawn not server-side %n for numbering #794 (pda)
• IMDSv2Tokens parameter: optional / required #786 (holmesjr) → #788 & #789 (pda)

Changed

• Default to gp3 volumes, previously gp2 #784 (yob)

Fixed

• c6gn.* instances recognized as ARM #785 (yob)
• s3secrets-helper installation more resilient #783 (shevaun)

Upgrading

Perform a CloudFormation stack update with the following URL:
https://s3.amazonaws.com/buildkite-aws-stack/v5.2.0/aws-stack.yml

If you want to launch a new stack, you can use this link (make sure not to use your production AWS account, create a new one for CI):

Documentation

See the Readme for this release.

Elastic CI Stack for AWS v5.1.0

Added

• Experimental support for ARM instance types (linux only) #758 (yob)
• Support up to four instance types and mixed combinations of Spot/OnDemand instances #710 (yob)
  • The InstanceType stack parameter can now be a CSV with up to 4 types
  • The new OnDemandPercentage stack parameter can be reduced from 100% (the default) to allow some Spot instances

Changed

• Update Buildkite Agent to v3.26.0 #778 (JuanitoFatas)
• Speed up secret downloads from S3 (from ~8 seconds to under 1 second) #772 (pda)
• ECR plugin now has its own log group header to make run time visible #773 (pda)

Fixed

• Avoid IAM changes for some kinds of stack updates (like changing InstanceType) #781 (yob)
• Improved documentation
  • Add BUILDKITE_PLUGIN_S3_SECRETS_BUCKET_PREFIX to README #775 (maatthc)
  • Remove outdated advice re AgentsPerInstance #760 (niceking)

Upgrading

Perform a CloudFormation stack update with the following URL:
https://s3.amazonaws.com/buildkite-aws-stack/v5.1.0/aws-stack.yml

If you want to launch a new stack, you can use this link (make sure not to use your production AWS account, create a new one for CI):

Documentation

See the Readme for this release.

Elastic CI Stack for AWS v5.0.1

This is a bugfix release relevant if you're using Windows and want to use AWS SSM Parameter Store for storing your agent token. Please see the v5.0.0 release for full notes on what's new in version 5 of the stack.

Fixed

• Retrieve agent token from parameter store on Windows agents #762 (chrisfowles)

Upgrading

In most cases, you should be able to upgrade in-place older versions of the stack with a CloudFormation stack update using the following template URL:
https://s3.amazonaws.com/buildkite-aws-stack/v5.0.1/aws-stack.yml

If you want to launch a new stack, you can use this link (make sure not to use your production AWS account, create a new one for CI):

Documentation

See the Readme for this release.

Elastic CI Stack for AWS v5.0.0

What's new?

Previously experimental Lambda-based scaler is the new default 🚀

Our previously experimental fast autoscaler that scales up much faster is now always used. Instances will automatically scale to demand using the stack defaults and you should see good results without changing any parameters.

Experimental Windows support is now available

A new AMI built on Windows Server 2019 is optionally available to use in the stack for Windows builds. You can enable this by changing the InstanceOperatingSystem setting from the default linux to windows. This release includes recent updates to Windows stability on Buildkite agent v3.25.0. Huge thanks to @jeremiahsnapp and @tduffield for their contributions here!

Note: There is a known issue with graceful handling of spot instances under windows. The agent may not disconnect gracefully, and may appear in the Buildkite UI for a few minutes after they terminate (See issue #752). We recommend using Windows on-demand instances for now.

Added

• Support for buildkite/image-builder which can enable you to customize AMIs based off the ones we ship #692 (keithduncan)
• Support for multiple security groups on instances #667 (jdub)
• AMI and Lambda Scaler support more regions: ap-east-1 (Hong Kong), me-south-1 (Bahrain), af-south-1 (Cape Town), eu-south-1 (Milan) #718 (JuanitoFatas)
• Support for loading BuildkiteAgentTokenPath from AWS Parameter Store #601 (jradtilbrook), #625 (jradtilbrook)

Changed

• Docker configuration is now isolated per-step #678 (patrobinson) #756 (yob)
• Use EC2 LaunchTemplate instead of a LaunchConfiguration #589 (lox)
• InstanceType default is now t3.large (was t2.nano) #699 (pda)
• Disable AZRebalancing to prevent running instances being terminated unnecessarily #751

Dependencies updated

• Bump Buildkite Agent to v3.25.0 #726 (JuanitoFatas)
• Bump Buildkite Agent Scaler to v1.0.2 #724 (JuanitoFatas) 4fafd8e (JuanitoFatas)
• Bump docker to v19.03.13 (linux) and v19.03.12 (windows) and docker-compose to v1.27.4 (linux, windows uses latest choco version) #719 (yob) #723 (JuanitoFatas)
• Bump bundled plugins to the latest versions: secrets, ecr and docker login

And much, much more, we recommend you view the full changelog for this release.

Upgrading

In most cases, you should be able to upgrade in-place older versions of the stack with a CloudFormation stack update using the following template URL:
https://s3.amazonaws.com/buildkite-aws-stack/v5.0.0/aws-stack.yml

If you want to launch a new stack, you can use this link (make sure not to use your production AWS account, create a new one for CI):

If you were using them, the following parameters have been removed or reworked:

• EnableExperimentalLambdaBasedAutoscaling was removed (it's the default now)
• BuildkiteOrgSlug was removed – the information reported by buildkite-agent-scaler make it redundant, but consider buildkite-agent-metrics if you need more detailed metric monitoring that supports multiple metric backends
• BuildkiteTerminateInstanceAfterJobTimeout, ScaleDownPeriod and ScaleCooldownPeriod are replaced by the more concise ScaleInIdlePeriod #586 (jeremiahsnapp) which lets agents self-terminate to scale in when they've been idle after a set period
• BuildkiteTerminateInstanceAfterJobDecreaseDesiredCapacity and ScaleDownAdjustment were removed - instances will now always try to decrement the ASG desired count when their waiting period for new jobs has elapsed
• ScaleUpAdjustment is replaced by ScaleOutFactor as the new lambda scaler calculates how many agents are needed at the time. ScaleOutFactor allows you to multiply the relative quantity provisioned to the ASG, either to slow it down or speed it up

Documentation

See the Readme for this release.

Elastic CI Stack for AWS v5.0.0 Beta 1

This is a beta release. Most things should work, but it's possible there are undiscovered issues in the release and we recommend continuing to use our stable v4.5.0 release for critical workloads. If you do spot any problems, we'd really appreciate it if you submit a GitHub issue or reply to our forum thread for this beta release where we will offer continued support.

What's new?

Previously experimental Lambda-based scaler is the new default 🚀

Our previously experimental fast autoscaler that scales up much faster is now always used. Instances will automatically scale to demand using the stack defaults and you should see good results without changing any parameters.

Experimental Windows support is now available

A new AMI built on Windows Server 2019 is optionally available to use in the stack for Windows builds. You can enable this by changing the InstanceOperatingSystem setting from the default linux to windows. We've observed issues with Windows instances when MinSize is set above 0, so please avoid that for now. Huge thanks to @jeremiahsnapp and @tduffield for their contribution here!

Added

• Support for buildkite/image-builder which can enable you to customize AMIs based off the ones we ship #692 (keithduncan)
• Support for multiple security groups on instances #667 (jdub)
• AMI and Lambda Scaler support more regions: ap-east-1 (Hong Kong), me-south-1 (Bahrain), af-south-1 (Cape Town), eu-south-1 (Milan) #718 (JuanitoFatas)
• Support for loading BuildkiteAgentTokenPath from AWS Parameter Store #601 (jradtilbrook), #625 (jradtilbrook)

Changed

• Use EC2 LaunchTemplate instead of a LaunchConfiguration #589 (lox)
• InstanceType default is now t3.large (was t2.nano) #699 (pda)

Dependencies updated

• Bump Buildkite Agent to v3.24.0 #726 (JuanitoFatas)
• Bump Buildkite Agent Scaler to v1.0.1 #724 (JuanitoFatas)
• Bump docker to v19.03.13 (linux) and v19.03.12 (windows) and docker-compose to v1.27.4 (linux, windows uses latest choco version) #719 (yob) #723 (JuanitoFatas)

And much, much more, we recommend you view the full changelog for this release.

Upgrading

Perform a CloudFormation stack update with the following URL:
https://s3.amazonaws.com/buildkite-aws-stack/v5.0.0-beta1/aws-stack.yml

If you want to launch a new stack, you can use this link (make sure not to use your production AWS account, create a new one for CI):

Documentation

See the Readme for this release.

Elastic CI Stack for AWS v4.5.0

Changed

• Added ImageIdParameter CloudFormation parameter for SSM Parameter Store image lookup #691 (@keithduncan)

Upgrading

Perform a CloudFormation stack update with the following URL:
https://s3.amazonaws.com/buildkite-aws-stack/v4.5.0/aws-stack.yml

If you want to launch a new stack, you can use this link (make sure not to use your production AWS account, create a new one for CI):

Documentation

See the Readme for this release.