test: integration tests for the tier 1 release fixes

[NickJosevski]

Adds end-to-end integration tests for the four "Tier 1" fixes — the ones whose correctness depends on how the real Octopus Server behaves, which testutil.MockHttpServer cannot validate by construction.

Refs #294, #426, #250, #556.

Why this branch stacks the four fixes

The tests assert post-fix behaviour, so they need the fixes present to pass. This branch merges the four feature branches and adds the tests on top:

Merged	PR
nj/issue-294	#696
nj/issue-426	#695
nj/issue-250	#702
nj/issue-556	#692

This branch is not for merging as-is. It exists to prove the four compose and to carry the new tests. Once the four land on main, the last commit here rebases onto main on its own.

Tests added

All in test/integration/release_test.go, following the existing harness (integration.RunCli, CreateCommonProject, t.Cleanup teardown).

• TestReleaseDeployUnknownVersion — release deploy returned error when using latest as input param for version #294. Asserts the API premise directly (an unknown version yields no usable release), then that release deploy names the version it could not find, and that latest is explained rather than passed through.
• TestReleaseCreateMissingPackageVersion — Unhelpful output when attempting to create a release with a package that doesn't exist #426. Project with a package step whose package has no version in the built-in feed; asserts the failure names the package and the step.
• TestReleaseCreateAndDeployByID — Some arguments do not accept the ID instead of the name #250. release create --channel <Channels-N> lands on that channel; release deploy --environment <Environments-N> produces a deployment in that environment.
• TestReleaseDeployCommaSeparatedTargets — Support comma-delimited values on octopus release deploy --deployment-target command #556. Two cloud-region targets, --deployment-target "A,B", asserts the resulting deployment's SpecificMachineIds contains both.

Verification

Run against a real Octopus Server (local dev instance, server main):

• On this branch: all 4 pass, 20.7s, clean teardown.
• On unmodified main with the same test file: all 4 fail. They are genuine regression tests, not tests that pass either way.
• go build ./... clean; go test ./pkg/... green (63 packages).

Finding: the null-reference symptom no longer reproduces

#294 and #426 both describe Octopus API error: Object reference not set to an instance of an object. []. On a current server that is not what happens:

• release deploy returned error when using latest as input param for version #294 path → Octopus API error: Release 9.9.9 for project <name> was not found. []
• Unhelpful output when attempting to create a release with a package that doesn't exist #426 path → Octopus API error: There are no viable release plans in any channels using the provided arguments... Cannot resolve

The server-side defect appears to have been fixed since those issues were filed (2022.3 and 2024.4 respectively). Both CLI fixes still improve the message materially, and older servers still exhibit the original behaviour — but the premise that the server null-refs is no longer true on current versions. Two consequences:

1. The suggestion in fix: report missing package versions instead of a server null reference #695 to raise a matching Server issue is probably moot; worth confirming before filing.
2. The assert.NotContains(..., "Object reference not set") lines in these tests are not load-bearing on a current server. They are kept as regression guards for older ones; the positive assertions are what carry the tests.

Finding: the four fixes do not compose without test changes

Each of the four is green on its own branch, but merged they break each other's unit tests — invisible on the individual branches by construction. Fixed in the first commit here:

• pkg/cmd/release/deploy/deploy_test.go, 8 conflict hunks: release deploy returned error when using latest as input param for version #294 adds a release pre-flight lookup and Some arguments do not accept the ID instead of the name #250 adds an environment lookup to the same request sequences. Resolved by expecting both, release first, matching the merged resolution order in deployRun.
• The tests added by Some arguments do not accept the ID instead of the name #250 and Support comma-delimited values on octopus release deploy --deployment-target command #556 merged cleanly but had no release pre-flight expectation, so MockHttpServer deadlocked waiting for a request that never came — the full unit suite hung rather than failed.
• Some arguments do not accept the ID instead of the name #250's release deploy specifying project, environment and tenant by ID still expected the two post-deploy web-URL lookups that release deploy returned error when using latest as input param for version #294 removes.
• Support comma-delimited values on octopus release deploy --deployment-target command #556's runbook comma-separated test needed Some arguments do not accept the ID instead of the name #250's environments/all lookup.

None of these are defects in the individual PRs; they are ordinary merge fallout. Flagging them because whichever of the four merges last will hit exactly this, and the failure mode for two of them is a hang, not a red test.

Notes

• test/integration has no build tag and GetApiClient calls os.Exit(999) when OCTOPUS_TEST_URL/OCTOPUS_TEST_APIKEY are unset, so a bare go test ./... from the repo root hard-exits. Run the suite from test/integration.
• The two deploy tests queue a real server task and wait for it to complete, because a project cannot be deleted while one is running. Whether the deployment succeeds is not asserted.
• allowDeploymentsTo restores the fixture lifecycle's phases on cleanup, otherwise the environment cannot be deleted.

🤖 Generated with Claude Code

[NickJosevski]

Nil pointer dereference on the post-create lookup failure path (pre-existing, but this function is being touched here and the new diagnosis flow makes create failures more visible): a few lines below at the options.Response handling, when octopus.Releases.GetByID(options.Response.ReleaseID) fails, the error branch still dereferences the nil result:

newlyCreatedRelease, lookupErr := octopus.Releases.GetByID(options.Response.ReleaseID)
if lookupErr != nil {
    cmd.PrintErrf("Warning: cannot fetch release details: %v\n", lookupErr)
    printReleaseVersion(options.Response.ReleaseVersion, newlyCreatedRelease.Assembled, newlyCreatedRelease.ReleaseNotes, nil)

ReleaseService.GetByID returns nil, err on failure, so a transient server error right after a successful create panics the CLI instead of printing the warning.

[NickJosevski]

The package diagnosis can misattribute an unrelated 5xx and hide the real cause. This branch runs for any 5xx, not just the null-reference case, and MissingPackageVersionsError.Error() does not include the original server message (it is only reachable via Unwrap). If the server 500s for an unrelated reason (timeout, genuine server bug) while the project happens to contain a package with no version in its feed — or the CLI's re-derived baseline disagrees with the server (e.g. a --package override the CLI silently failed to parse but the server accepted) — the user is told to push packages instead of seeing the actual failure.

Consider gating the missing-package diagnosis on strings.Contains(apiError.ErrorMessage, serverNullReferenceMessage) as well, and/or including the wrapped cause text in the error output.

[NickJosevski]

Regression for names that legitimately contain commas, with no escape hatch. Octopus allows commas in environment/tenant/machine names. Before this change --environment "Dev, East" (one env named Dev, East) worked; now it is unconditionally split into Dev + East and the deploy fails with "cannot find an environment...". The doc comment acknowledges the constraint but there is no way for a user to opt out (quoting doesn't help — the split happens after shell parsing).

The old octo CLI had the same splitting behaviour, so this may be an accepted trade-off — but worth an explicit decision and a mention in the flag help/changelog, since previously-working invocations now break silently.

[NickJosevski]

False "cannot find a tenant" for tenants beyond the first page of a partial-name search. The SDK's Tenants.GetByIdentifier name fallback (GetByName) issues Get(TenantsQuery{PartialName: name}) and scans only the first page of results for an exact match — it never pages. On a space with many tenants whose names share a common substring (e.g. dozens of "Store ..." tenants), a tenant whose exact name lands beyond page 1 returns ErrItemNotFound, and this new resolution step fails a deploy/runbook run that previously worked (the raw name used to be passed straight to the executions API, which matched it fine).

Also note GetByIdentifier uses a direct type assertion on the GetByID error, so a non-APIError failure (network blip) silently falls through to the name lookup rather than being reported.

[NickJosevski]

A mixed list of regular + ephemeral environment identifiers can never resolve. selectors.FindEnvironments errors on the first identifier that isn't a regular environment, and the fallback findEphemeralEnvironments errors on the first identifier that isn't ephemeral — so --environment regularEnv,ephemeralEnv now fails client-side with "cannot find an environment ..." even though before this change both names were passed through verbatim for the server to judge. Probably invalid server-side anyway (one channel type per deployment), but if so the current error message points at the wrong thing: it claims the regular env doesn't exist when it does. Resolving each identifier individually (regular-then-ephemeral per item) would handle both this and give a precise error.

[NickJosevski]

Silent precedence flip for existing callers. The old executionscommon.FindEnvironments checked the name lookup before the ID lookup; this new implementation checks ID first. Since executionscommon.FindEnvironments is now an alias to this, the change reaches all its existing callers (tenant connect, five target ... create commands, runbook, deploy): in a space where an environment is named the same as another environment's ID, those commands now resolve to a different environment than before. The tests show this is deliberate ("consistent with how projects and tenants resolve") — flagging it because it's an observable behaviour change to commands this PR doesn't otherwise touch, and may deserve a changelog note.

[NickJosevski]

Every automation-mode invocation now pays extra HTTP round trips even when plain names were given. For each tenant supplied by name this is two requests (a guaranteed 404 on GET /tenants/<name>, then the partial-name search), plus GET /environments/all, plus the release lookup — sequentially, on every CI deploy. The same pattern is in runbook run (and there the environment/tenant resolution also runs in interactive mode, where AskQuestions resolves environments again — duplicate /environments/all calls).

A cheap win: only hit the ID lookup when the identifier actually looks like an ID (^Tenants-\d+$ / ^Environments-\d+$), or resolve tenants with a single query instead of per-tenant round trips. Related: selectors.FindEnvironment (singular) previously used a paged partial-name server query and now loads the entire space's environment list to find one environment.

[NickJosevski]

Altitude: the ID-to-name resolution is scattered per command, at differing depths. release create resolves the channel only in the automation branch; release deploy resolves tenants before both modes but environments only in the automation branch; runbook run resolves both unconditionally. The invariant they all enforce ("the executions API only matches by name") belongs to the layer that builds the executions-API commands (pkg/executor/release.go / the runbook executor), where one implementation would cover all three commands, both modes, and any future execution flag — instead of a pattern that has to be remembered (and is already applied inconsistently) in each command. Fine to land as-is for the tier-1 fixes, but worth a follow-up.