Pod hosting Remote Agent doesn't become Ready

[vadiml77]

Describe the bug

The pod with the remote agent does get deployed, however the pod is not Ready from the K8S perspective. Consequently the service being debugged never switches to the pod hosting the agent, and continues to route to the original pod

Logs

[~] $ k logs ssp-dev-ssp-auth-mgr-67c675b7cb-w6sfc
2022-03-13T15:48:24.8934845Z | RemoteAgent | TRACE | SyncService.RunTar tar succeeded. \nOperation context: {"clientRequestId":"721219b1-ac86-4e2e-a3ab-966360fb219e","correlationId":"547604da-2483-4997-b66e-767db40684b71647186379555:5c65c88fef4b:d568409433ec","requestId":null,"userSubscriptionId":null,"startTime":"2022-03-13T15:48:17.5916105Z","userAgent":"RemoteAgent/1.0.0.0","requestHttpMethod":null,"requestUri":null,"version":"1.0.0.0","requestHeaders":{},"loggingProperties":{"applicationName":"RemoteAgent","deviceOperatingSystem":"Linux 3.10.0-1160.11.1.el7.x86_64 #1 SMP Fri Dec 18 16:34:56 UTC 2020","framework":".NET Core 3.1.15","processId":1,"targetEnvironment":"Production"}}
2022-03-13T15:48:25.5501874Z | RemoteAgent | TRACE | SyncService.RunTar tar succeeded. \nOperation context: {"clientRequestId":"721219b1-ac86-4e2e-a3ab-966360fb219e","correlationId":"547604da-2483-4997-b66e-767db40684b71647186379555:5c65c88fef4b:d568409433ec","requestId":null,"userSubscriptionId":null,"startTime":"2022-03-13T15:48:17.5916105Z","userAgent":"RemoteAgent/1.0.0.0","requestHttpMethod":null,"requestUri":null,"version":"1.0.0.0","requestHeaders":{},"loggingProperties":{"applicationName":"RemoteAgent","deviceOperatingSystem":"Linux 3.10.0-1160.11.1.el7.x86_64 #1 SMP Fri Dec 18 16:34:56 UTC 2020","framework":".NET Core 3.1.15","processId":1,"targetEnvironment":"Production"}}
2022-03-13T15:48:26.0771109Z | RemoteAgent | TRACE | SyncService.RunTar tar succeeded. \nOperation context: {"clientRequestId":"721219b1-ac86-4e2e-a3ab-966360fb219e","correlationId":"547604da-2483-4997-b66e-767db40684b71647186379555:5c65c88fef4b:d568409433ec","requestId":null,"userSubscriptionId":null,"startTime":"2022-03-13T15:48:17.5916105Z","userAgent":"RemoteAgent/1.0.0.0","requestHttpMethod":null,"requestUri":null,"version":"1.0.0.0","requestHeaders":{},"loggingProperties":{"applicationName":"RemoteAgent","deviceOperatingSystem":"Linux 3.10.0-1160.11.1.el7.x86_64 #1 SMP Fri Dec 18 16:34:56 UTC 2020","framework":".NET Core 3.1.15","processId":1,"targetEnvironment":"Production"}}
2022-03-13T15:48:27.5501079Z | RemoteAgent | TRACE | ReversePortForwardConnector created for port 8086\nOperation context: {"clientRequestId":"721219b1-ac86-4e2e-a3ab-966360fb219e","correlationId":"547604da-2483-4997-b66e-767db40684b71647186379555:5c65c88fef4b:d568409433ec","requestId":null,"userSubscriptionId":null,"startTime":"2022-03-13T15:48:17.5916105Z","userAgent":"RemoteAgent/1.0.0.0","requestHttpMethod":null,"requestUri":null,"version":"1.0.0.0","requestHeaders":{},"loggingProperties":{"applicationName":"RemoteAgent","deviceOperatingSystem":"Linux 3.10.0-1160.11.1.el7.x86_64 #1 SMP Fri Dec 18 16:34:56 UTC 2020","framework":".NET Core 3.1.15","processId":1,"targetEnvironment":"Production"}}
2022-03-13T15:48:27.5556324Z | RemoteAgent | TRACE | ReversePortForwardConnector start listening on port 8086
2022-03-13T15:48:41.7832360Z | RemoteAgent | TRACE | ReversePortForwardConnector on port 8086 accepted incoming request as stream 1.
2022-03-13T15:48:41.7928172Z | RemoteAgent | TRACE | AgentHub connnected for 8086, id 1
2022-03-13T15:48:41.7940706Z | RemoteAgent | TRACE | ReversePortForwardConnector.HookupStreamData ReadAsync 1 returns 237 bytes.
2022-03-13T15:48:41.7988469Z | RemoteAgent | TRACE | AgentHub received for 8086, id 1, size 237
2022-03-13T15:48:41.9526498Z | RemoteAgent | TRACE | AgentHub disconnect for 8086, id 1\nOperation context: {"clientRequestId":"721219b1-ac86-4e2e-a3ab-966360fb219e","correlationId":"547604da-2483-4997-b66e-767db40684b71647186379555:5c65c88fef4b:d568409433ec","requestId":null,"userSubscriptionId":null,"startTime":"2022-03-13T15:48:17.5916105Z","userAgent":"RemoteAgent/1.0.0.0","requestHttpMethod":null,"requestUri":null,"version":"1.0.0.0","requestHeaders":{},"loggingProperties":{"applicationName":"RemoteAgent","deviceOperatingSystem":"Linux 3.10.0-1160.11.1.el7.x86_64 #1 SMP Fri Dec 18 16:34:56 UTC 2020","framework":".NET Core 3.1.15","processId":1,"targetEnvironment":"Production"}}
2022-03-13T15:48:41.9552076Z | RemoteAgent | TRACE | PortForwardConnector.Disconnect 1
2022-03-13T15:48:41.9566303Z | RemoteAgent | TRACE | AgentHub closed for 8086, id 1
2022-03-13T15:48:41.9837605Z | RemoteAgent | TRACE | ReversePortForwardConnector.StartReceiveDataAsync exception 'Unable to read data from the transport connection: Operation canceled.' when invoking handler. Close.

Describe Pod logs for the pod with remote agent

Events:
Type Reason Age From Message

---

Normal Scheduled Successfully assigned dev/ssp-dev-ssp-auth-mgr-67c675b7cb-w6sfc to HOSTNAME_REDACTED
Normal Pulling 14m kubelet, HOSTNAME_REDACTED Pulling image "lolodi/devhostagent:20210909.1"
Normal Pulled 14m kubelet, HOSTNAME_REDACTED Successfully pulled image "lolodi/devhostagent:20210909.1" in 934.946905ms
Normal Created 14m kubelet, HOSTNAME_REDACTED Created container ssp-auth-mgr
Normal Started 14m kubelet, HOSTNAME_REDACTED Started container ssp-auth-mgr
Warning Unhealthy 11m (x3 over 13m) kubelet, HOSTNAME_REDACTED Liveness probe failed: Get "https://10.39.0.11:8086/auth/v1/health/liveness": EOF
Warning Unhealthy 4m41s (x55 over 14m) kubelet, HOSTNAME_REDACTED Readiness probe failed: Get "https://10.39.0.11:8086/auth/v1/health/readiness": EOF