Improve end customer testing of acquiring packages that ship as beta, combined with other packages that are GA.

[ahsonkhan]

It is important for customers to be able to adopt our specific beta packages, easily, so they can provide valuable feedback on new features, while keeping the rest of the dependency graph stable.

For example, have a test job (running in CI/nightly) where the customer can successfully build an application using a GA'd version of azure-core-cpp, with a beta version of azure-identity-cpp (or any other beta client package), acquiring them via vcpkg.

Such end-user package acquisition testing will help catch build issues that require patching the beta registry.
An example of that was here:
Azure/azure-sdk-vcpkg-betas@ad5bdf2
#6042

Currently, our CI pipeline tests and verifies all sources in main build well together, and the CI in the vcpkg main registry tests to ensure all GA'd packages work well. But there's a gap where beta packages are mixed with GA'd packages.

A way to simulate that scenario and repro the issue is by building an application which only contains the specific beta package being tested in its vcpkg-configuration.json, and not the entire list of packages we ship (for example, filtering this down to just azure-identity-cpp results in vcpkg getting the azure-core-cpp dependency from the GA registry):
https://github.com/Azure/azure-sdk-vcpkg-betas/blob/62c56807f81f0cae7c6ee97a3eb4edc79f2c5481/vcpkg-configuration.json#L13-L30

cc @RickWinter, @ronniegeraghty

[LarryOsterman]

I would go well beyond this and have stages in the nightly pipeline which test:

1. Building a sample application in the repo that uses a representative set of functionality for each package. This sample would pull from the main vcpkg registry using current GA bits.
2. Building the same sample applications in the repo using the beta registry.

Note that in each case, a powershell script should be used on a clean windows-2022/Ubuntu 22.04 image which:

1. Acquires vcpkg
2. Configures vcpkg to use the beta registry if needed
3. Builds the sample

The key part here is that we have an artifact checked into our tree which automates the acquisition process for clients of the SDK. That way customers have an example of live code that performs the acquisition and NOT documentation online.

[ahsonkhan]

Absolutely!

One subtlety that I do want to emphasize, along with what you shared Larry, is that the mix of beta and GA packages tested together is important here. Not just only GAs or only betas, separately (which we already have some testing of those, in silos). Using the identity beta with the core GA is an example of what uncovered the types of build issues we saw here (using core beta wouldn't have hit it).

The automation you called out is also an important aspect to this.