As a package author, I should be able to GA release individual packages on vcpkg with confidence by having automated testing that detects potential build errors earlier based on package permutations

[ahsonkhan]

The problem:

• We don't always GA or release all packages at the same time/milestone. Therefore, we need a mechanism of testing of shipping individual packages prior to the day we actually ship. This is to detect build errors early based on a combinatorics of shipping possibilities, rather than discovering them at the day of release. Given our dependency tree within the SDK is relatively flat, this largely means being able to test shipping a GA of azure-core out of main (which all upstream packages depend on), and validating whether that would pass when we don't intend to ship some subset of other upstream packages.

An example of the problem we experienced:

• We made changes to azure-core and used it in upstream packages (including eventhubs as an example). Our daily CI and vcpkg package testing, based on main, all pass. When we went to ship a GA of azure-core (and only azure-core, since we didn't intend to ship some of the upstream packages like eventhubs, keyvault, etc.), the existing published package in the vcpkg GA registry failed to build, blocking the release of core. Note, the individual changes were all reasonable, so the onus is on systems in place to detect what could break, to help the team as the last line of defense.

• To fix this, we had to patch azure-core-cpp and all upstream packages that depended on it which we didn't intend to ship in that release cycle (when we discovered the issue, on release day). The problem is less so that we hit this issue (which can happen again and easily fixed, whenever changes are made to core that upstream packages depend on), but rather, when we hit it in the release cycle:

  • UserAgentGenerator compatibility for previously published EventHubs #6063
  • https://github.com/microsoft/vcpkg/pull/41321/files#diff-d4ea6df76b5552779a4f55327e576eea395fe80997b4ed706e8e4be9d2110135R1

Where we are today:

• We currently have CI/live daily testing for all PRs and code check-ins in main. These tests all the packages, essentially, together (almost like a mono repo).
• We also have a standing PR in vcpkg which runs the package validation (daily) simulating what would happen if we were to ship GAs of our SDKs at any given time, allowing us to get a heads up early if something fails. That's the intention, but it didn't catch this issue (and was showing as green) since it assumes we are shipping everything and checks if building all packages together would succeed. That's certainly useful for some scenarios, but is not sufficient in fulfilling the whole intent.

Where we'd like to be:

• As the permutations of the packages we ship grows, at various stability levels (which is reasonable and intended by design), we'd like to get early notice of any potential and unintentional build breaks. To achieve this, we need testing of the specific permutations of packages we plan to ship at a particular milestone and how they'd interact with existing shipped packages that won't be updated, well before the day we actually release the subset of packages we'd want to GA to the main vcpkg registry (which includes pre-first-GA betas).

P.S. This issue is related to the beta registry package testing, but different since it focuses on testing and shipping of individual packages as GA to the main vcpkg registry (and not about mixing packages of different dev stages).

cc @RickWinter, @ronniegeraghty, @danieljurek

[danieljurek]

In practice there are a couple of "core" packages (core, identity) and the tests we want to run are with vcpkg released GA versions and what's in source. This would be 4N complexity (excluding packages like those in storage that depend on storage-common).

If using the 4N approach, this could take the form of jobs which build a sample project with core and identity from current state of the repo and from vcpkg.