Dev Spaces Service Health Dashboard

[moewill]

Describe the bug
Running command azds space list fails with error message: The operation was canceled.

To Reproduce
az aks get-credentials -g <rg-name> --name <cluster-name>
azds controller list
azds controller select -g <rg-name> -n <controller-name>
azds space list -v

A list of the existing Dev Spaces is returned with the appropriate column headers.

Logs
2020-05-19T16:15:22.2951820Z | CLI | TRACE | Event: Command.Start <json>{"properties":{"arguments":"space list -v"},"metrics":null}</json>\nOperation context: <json>{"clientRequestId":null,"correlationRequestId":null,"requestId":null,"userSubscriptionId":"<subscription-id>","startTime":"2020-05-19T16:15:21.9678980Z","userAgent":"CLI/1.0.20200507.16","requestHttpMethod":null,"requestUri":null,"apiVersion":"3.2","version":"1.0.20200507.16","requestHeaders":{},"loggingProperties":{"applicationName":"CLI","deviceOperatingSystem":"Darwin 19.4.0 Darwin Kernel Version 19.4.0: Wed Mar 4 22:28:40 PST 2020; root:xnu-6153.101.6~15/RELEASE_X86_64","framework":".NET Core 4.6.26515.07","macAddressHash":"<mac-hash>","commandId":"64a842ba-5f71-4140-af80-32ea9b2ecef8","processId":10055,"targetEnvironment":"Production"}}</json>

Environment Details
Client used: CLI

1.0.20200507.16
API v3.2```

Client's version: `API v3.2`

Operating System: `Mac OS`

I was informed also that the product team is reviewing the issue, so insight on when the service is down or active tickets is good to know so that development teams are informed and can adjust if needed.

[amsoedal]

Hi @moewill, does this reproduce consistently? Usually "the operation was cancelled" indicates a transient issue, but I'm curious to know if you're seeing it every time.

[moewill]

Yes @amsoedal , unfortunately, I am seeing it every time...

[amsoedal]

@moewill would you be able to attach more of your logs? A request id or a subscription id would be helpful for us to investigate on our side. Thank you!

[moewill]

Thank you @amsoedal

The request IDs are all null:

2020-05-19T16:15:22.2951820Z | CLI | TRACE | Event: Command.Start <json>{"properties":{"arguments":"space list -v"},"metrics":null}</json>\nOperation context: <json>{"clientRequestId":null,"correlationRequestId":null,"requestId":null,

Can I PM you the subscription ID?

[amsoedal]

Yes! that would be great. You can email me at ansoedal@microsoft.com

[moewill]

Awesome - just sent 👍

[amsoedal]

Thanks @moewill, investigating now. Will follow up on email.

[moewill]

OK great, thank you!

…

-- Maurice Williams DevOps Engineer maurice@flux7.com Confidentiality Note: This e-mail and any attachments are confidential and may be protected by legal privilege. If you are not the intended recipient, be aware that any disclosure, copying, distribution or use of this e-mail or any attachment is prohibited. If you have received this e-mail in error, please notify us immediately by returning it to the sender and delete this copy from your system. Thank you for your cooperation.

On May 19, 2020, at 2:42 PM, amsoedal ***@***.***> wrote: Thanks @moewill <https://github.com/moewill>, investigating now. Will follow up on email. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#342 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALSGMF5V66VNF5DU5TJTIO3RSLHJFANCNFSM4NFFCYHA>.

-- Confidentiality Note: This e-mail and any attachments are confidential and may be protected by legal privilege. If you are not the intended recipient, be aware that any disclosure, copying, distribution or use of this e-mail or any attachment is prohibited. If you have received this e-mail in error, please notify us immediately by returning it to the sender and delete this copy from your system. Thank you for your cooperation.

[keikumata]

@moewill This seems to be an issue pertaining to the API server authorized IP ranges feature. Please take a look at https://github.com/Azure/dev-spaces/tree/master/public-ips to allow additional ranges based on your region when using AKS with Dev Spaces.

This is documented here: https://docs.microsoft.com/en-us/azure/aks/api-server-authorized-ip-ranges#create-an-aks-cluster-with-api-server-authorized-ip-ranges-enabled

[moewill]

Thanks Kei Yoshikoshi, We actually hit this issue last week with working with Davion Lowe and Micael Noto. The GitHub list is incorrect, which we discovered when hitting an issue with provisioning the Dev Space controller. Here is a brief sequence of events: - Controller in failed provisioning state - East US whitelist ranges added from https://github.com/Azure/dev-spaces/blob/master/public-ips/EastUS.txt <https://github.com/Azure/dev-spaces/blob/master/public-ips/EastUS.txt> - Controller provisioning failed again - Removed whitelist - Controller provisioning succeeded - Re-enabled whitelisting, including the above ranges and what’s listed here https://www.microsoft.com/en-us/download/details.aspx?id=56519 <https://www.microsoft.com/en-us/download/details.aspx?id=56519>, namely: { "name": "AzureDevSpaces.EastUS", "id": "AzureDevSpaces.EastUS", "properties": { "changeNumber": 1, "region": "eastus", "platform": "Azure", "systemService": "AzureDevSpaces", "addressPrefixes": [ "20.42.6.32/27", "20.42.6.128/28", "20.42.64.64/26" ] } }, - Controller was able to be provisioned and status was “succeeded" Now with the controller provisioned, we arrive at this Github issue, where I am unable to use to create a new Dev Space or list existing spaces.

…

-- Maurice Williams DevOps Engineer maurice@flux7.com Confidentiality Note: This e-mail and any attachments are confidential and may be protected by legal privilege. If you are not the intended recipient, be aware that any disclosure, copying, distribution or use of this e-mail or any attachment is prohibited. If you have received this e-mail in error, please notify us immediately by returning it to the sender and delete this copy from your system. Thank you for your cooperation.

On May 19, 2020, at 9:21 PM, Kei Yoshikoshi ***@***.***> wrote: @moewill <https://github.com/moewill> This seems to be an issue pertaining to the API server authorized IP ranges feature. Please take a look at https://github.com/Azure/dev-spaces/tree/master/public-ips <https://github.com/Azure/dev-spaces/tree/master/public-ips> to allow additional ranges based on your region when using AKS with Dev Spaces. This is documented here: https://docs.microsoft.com/en-us/azure/aks/api-server-authorized-ip-ranges#create-an-aks-cluster-with-api-server-authorized-ip-ranges-enabled <https://docs.microsoft.com/en-us/azure/aks/api-server-authorized-ip-ranges#create-an-aks-cluster-with-api-server-authorized-ip-ranges-enabled> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#342 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALSGMF3LFA7QIBLLUY75EF3RSMWBRANCNFSM4NFFCYHA>.

-- Confidentiality Note: This e-mail and any attachments are confidential and may be protected by legal privilege. If you are not the intended recipient, be aware that any disclosure, copying, distribution or use of this e-mail or any attachment is prohibited. If you have received this e-mail in error, please notify us immediately by returning it to the sender and delete this copy from your system. Thank you for your cooperation.

[amsoedal]

Hi @moewill, after investigating some more my colleague figured out that this is connected to an ongoing infrastructure issue we're facing related to our whitelisted IPs. We are currently working on a release to get this fixed. Will keep this thread updated, but the fix should be out by EOD today or tomorrow, after which your cluster should be in a better state again

[moewill]

Thanks Amsoedal, Can you shed more light on the issue, and where in the future such an issue can be tracked? Was there a notice, email or service dashboard made available to impacted users where we could have been made aware? If it’s a config setting for such notifications, please let me know - thanks!

…

-- Maurice Williams DevOps Engineer maurice@flux7.com Confidentiality Note: This e-mail and any attachments are confidential and may be protected by legal privilege. If you are not the intended recipient, be aware that any disclosure, copying, distribution or use of this e-mail or any attachment is prohibited. If you have received this e-mail in error, please notify us immediately by returning it to the sender and delete this copy from your system. Thank you for your cooperation.

On May 20, 2020, at 4:27 PM, amsoedal ***@***.***> wrote: Hi @moewill <https://github.com/moewill>, after investigating some more my colleague figured out that this is connected to an ongoing infrastructure issue we're facing related to our whitelisted IPs. We are currently working on a release to get this fixed. Will keep this thread updated, but the fix should be out by EOD today or tomorrow, after which your cluster should be in a better state again — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#342 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALSGMF6MBJTB6BMMORHLFATRSQ4MVANCNFSM4NFFCYHA>.

-- Confidentiality Note: This e-mail and any attachments are confidential and may be protected by legal privilege. If you are not the intended recipient, be aware that any disclosure, copying, distribution or use of this e-mail or any attachment is prohibited. If you have received this e-mail in error, please notify us immediately by returning it to the sender and delete this copy from your system. Thank you for your cooperation.

[amsoedal]

Hi @moewill unfortunately we don't currently have a way of notifying on this granular level of outage. We're moving towards a local experience which won't require controllers and will thus have less potential for outages (you can read more here: #341). This new feature will provide similar capabilities to azds (debugging your code, routing). Hope this helps

[moewill]

Thank you @amsoedal

With that new feature, what will be the future of Dev Spaces currently? Will Dev Spaces and azds tooling be deprecated in favor of that new workflow, or will they both be available side by side?

This is helpful because adopting Dev Spaces, developing workflows and automation now will prove to be painful if needing to migrate to a completely new solution/implementation.

[amsoedal]

Our future investments currently revolve around the new workflow, but we'd love to hear your feedback on your current usage of Dev Spaces and what capabilities are most useful for you! Would you be open to scheduling a short (30 min) talk with us?

[moewill]

Thanks for the offer @amsoedal - we will likely plan to do that.

I also wanted to check on the status of the whitelisting issue?

[amsoedal]

Hi again @moewill ! The whitelisting issue should be fixed now, but I believe our on-call is currently dealing with a separate issue. Are you running into problems?

[moewill]

It has been turned off pending a response to this thread confirming it was resolved. Is there a way to confirm it has been resolved?

…

-- Maurice Williams DevOps Engineer maurice@flux7.com Confidentiality Note: This e-mail and any attachments are confidential and may be protected by legal privilege. If you are not the intended recipient, be aware that any disclosure, copying, distribution or use of this e-mail or any attachment is prohibited. If you have received this e-mail in error, please notify us immediately by returning it to the sender and delete this copy from your system. Thank you for your cooperation.

On May 28, 2020, at 1:25 PM, amsoedal ***@***.***> wrote: Hi again @moewill <https://github.com/moewill> ! The whitelisting issue should be fixed now, but I believe our on-call is currently dealing with a separate issue. Are you running into problems? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#342 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALSGMF4QU53KC6TIEFJHAC3RT2NCBANCNFSM4NFFCYHA>.

-- Confidentiality Note: This e-mail and any attachments are confidential and may be protected by legal privilege. If you are not the intended recipient, be aware that any disclosure, copying, distribution or use of this e-mail or any attachment is prohibited. If you have received this e-mail in error, please notify us immediately by returning it to the sender and delete this copy from your system. Thank you for your cooperation.

[amsoedal]

Hi @moewill, the issue is resolved. Please let us know if you encounter any issues!

[moewill]

Thank you @amsoedal .

Can you point me to the correct list of IP addresses that should be whitelisted for both the bare AKS service and also for the Dev Spaces feature?

[amsoedal]

For Dev Spaces:
[
"13.69.71.144/28",
"13.70.78.176/28",
"13.71.175.112/28",
"13.71.199.96/28",
"13.73.244.128/28",
"13.74.111.128/28",
"13.78.111.144/28",
"13.86.221.224/28",
"20.37.157.64/28",
"20.37.195.80/28",
"20.38.85.128/28",
"20.39.11.64/28",
"20.41.5.128/28",
"20.42.6.32/27",
"20.42.6.128/28",
"20.42.64.64/26",
"20.42.131.192/27",
"20.42.230.64/28",
"20.43.65.208/28",
"20.43.130.240/28",
"20.189.108.32/28",
"40.69.110.176/28",
"40.70.151.80/28",
"40.74.30.144/28",
"40.75.35.224/28",
"40.78.239.0/28",
"40.78.251.224/27",
"40.82.253.112/28",
"40.89.17.192/28",
"40.119.9.240/28",
"51.104.25.208/28",
"51.105.77.64/28",
"52.150.139.144/28",
"52.182.141.128/28",
"52.228.81.224/28",
"104.214.161.48/28"
]
(downloaded from https://www.microsoft.com/en-us/download/details.aspx?id=56519)

For AKS, see: https://docs.microsoft.com/en-us/azure/aks/limit-egress-traffic
Note: it's important to pass the client machine IP while creating the AKS cluster

By EOD (we're releasing this feature), you should be able to whitelist us using service tags as described here: https://github.com/Azure/dev-spaces/blob/master/public-ips/README.md

[rakeshvanga]

@moewill You can go over Azure Dev Spaces documentation for this specific scenario here: https://docs.microsoft.com/en-us/azure/dev-spaces/configure-networking#using-api-server-authorized-ip-ranges
As amsoedal, mentioned it would be updated to reflect service tags changes.

Also, the json list of ips downloaded from https://www.microsoft.com/en-us/download/details.aspx?id=56519 contains ips used by dev spaces per region as well, if you want to whitelist a small set of ip ranges.