Inconsistent credential resolution order in AWS .NET SDK compared to other SDKs and AWS CLI

[ppejovic]

We have GitLab Runners in EKS that run in pods leveraging IAM Roles for Service Accounts (IRSA). The IAM role associated with these pods is a least-privileged role that allows GitLab Runners to do things like call into s3 for cacheing dependencies, etc.

Higher-privileged AWS credentials required for executing infrastructure deployments (through Terraform) and integration tests (with .NET) are injected into the pipeline job environment as environment variables. We've noticed that in these pipelines, Terraform deployments (GO SDK?) and AWS CLI (Python SDK?) resolve the credentials from the environment variables, whereas .NET integration tests with the AWS .NET SDK fail because they are assuming the role of the pod service account.

Looking into the code it seems that the .NET SDK prioritizes Web Identity Token credentials (how IRSA is implemented) over Environment Variables, where as Go, Python and Java SDKs (at least) prioritize Environment Variables first:

.NET - Web Identity Token (line 54) resolved before Environment Variables (line 57):

aws-sdk-net/sdk/src/Core/Amazon.Runtime/Credentials/FallbackCredentialsFactory.cs

Lines 49 to 59 in 0305133

	CredentialsGenerators = new List<CredentialsGenerator>
	{
	#if BCL
	() => new AppConfigAWSCredentials(), // Test explicit keys/profile name first.
	#endif
	() => AssumeRoleWithWebIdentityCredentials.FromEnvironmentVariables(),
	// Attempt to load the default profile. It could be Basic, Session, AssumeRole, or SAML.
	() => GetAWSCredentials(credentialProfileChain),
	() => new EnvironmentVariablesAWSCredentials(), // Look for credentials set in environment vars.
	() => ECSEC2CredentialsWrapper(proxy), // either get ECS credentials or instance profile credentials
	};

Java SDK - Environment Variables resolved before Web Identity Token: https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html

GO SDK - Environment Variables resolved before Web Identity Token: https://aws.github.io/aws-sdk-go-v2/docs/configuring-sdk/#specifying-credentials

Python SDK - Environment Variables resolved before Web Identity Token (Assume Role Provider): https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials

This leads to inconsistent behaviour within pipelines that may be leveraging the AWS .NET SDK alongside tools using any other AWS SDK.

To me this seems like a bug in the .NET credential resolution order but I wanted to open a discussion first to see if there's a good justification for this behaviour being different from the other SDKs.

[ashishdhingra]

@ppejovic Please refer Standardized credential providers. It states that the distinct chain used by each SDK varies.. We have a related GitHub issue #1779 in our backlog, but it could be implemented in next major version. I will add a note about this discussion which could be reviewed by the team and/or implementor.

I will close this discussion for now given that we are already tracking similar issue.

Thanks,
Ashish