Powershell NetCore does not work with AWS_WEB_IDENTITY_TOKEN_FILE

[iguyking]

Description

Attempting to utilize AWS Powershell Netcore tooling with EKS using IRSA credentials which supply the AWS_WEB_IDENTITY_TOKEN_FILE environment variable and file for getting credentials for a kubernetes pod. When making calls such as Use-STSCallerIdentity, the AWS cmdlets do not find the Web Identity Token information. Instead it is defaulting to the node's instance profile.

Reproduction Steps

Create a basic Ubuntu Docker Image with .NET 5 (latest) and Powershell Core downloaded.
Install AWSPowershell.NetCore version 4.1.15.0.
Setup EKS with the recommended IRSA configuration and assign this role to the deployment pod in question
Run Get-STSCallerIdentity

This will return the node's IAM role, not the Pods.

AWS CLI will return correctly:

{
    "UserId": "AROAVHFETX5TATZNVXBVJ:botocore-session-1637098166",
    "Account": "123456789",
    "Arn": "arn:aws:sts::123456789:assumed-role/runner-service-account/botocore-session-1637098166"
}

Powershell call will return

{
  "Account": "123456789",
  "Arn": "arn:aws:sts::123456789:assumed-role/platform-2021102117024492730000000e/i-0c37ddbaf651488a1",
  "UserId": "AROAVHFETX5TH76JTXAAB:i-0c37ddbaf651488a1",
  "ResponseMetadata": {
    "RequestId": "b445fa1c-342a-477a-a2fb-c23b0e9a53d5",
    "Metadata": {}
  },
  "ContentLength": 493,
  "HttpStatusCode": 200,
  "LoggedAt": "2021-11-16T21:39:23.3531025+00:00"
}

Environment

• Build Version: 4.1.15.0
• OS Info: Ubuntu 20.04
• Build Environment: Download from nuget registry
• Targeted .NET Platform: 5

Resolution

• 👋 I can/would-like-to implement a fix for this problem myself

---

This is a 🐛 bug-report

[ashishdhingra]

Hi @iguyking,

Good morning.

Could you please confirm the following:

• Value of environment variable AWS_ROLE_ARN.
• Value of environment variable AWS_WEB_IDENTITY_TOKEN_FILE.
• EKS version

Some issues are resolved after cluster upgrade as was the case in aws/aws-sdk-net#1615.

Thanks,
Ashish

[iguyking]

EKS version is 1.20

AWS_ROLE_ARN & AWS_WEB_IDENTITY_TOKEN_FILE are set and work properly for the python based aws cli commands.

I ran the aws cli & powershell commands in the same OS/same docker container session.

That ticket is around Windows based OSes. This is from an ubuntu 20.04 image.

[iguyking]

What can I help give you to see this work or not work?

[iguyking]

Update:

EKS version 1.21.5-20211117
Powershell 7.2.0
Using Ubuntu 20.04.3 LTS.

Still not working where the aws python works.

PS /> import-module AWSPowershell.NetCore
PS /> gci env:A*

Name                           Value
----                           -----
AWS_ROLE_ARN                   arn:aws:iam::123423341127:role/powershell-test-role
AWS_WEB_IDENTITY_TOKEN_FILE    /var/run/secrets/eks.amazonaws.com/serviceaccount/token
AWS_DEFAULT_REGION             us-west-2
AWS_REGION                     us-west-2

PS /> Get-STSCallerIdentity | select *

LoggedAt         : 11/23/2021 22:54:09
Account          : 123423341127
Arn              : arn:aws:sts::123423341127:assumed-role/team2021111722262155740000000e/i-00e0324c06a60e198
UserId           : AXOAQWCVGYJDSCMQZGHOP:i-00e0324c06a60e198
ResponseMetadata : Amazon.Runtime.ResponseMetadata
ContentLength    : 486
HttpStatusCode   : OK


PS /> aws sts get-caller-identity
{
    "UserId": "AXOAQWCVGYJDTJRFCNTOS:botocore-session-1637708010",
    "Account": "123423341127",
    "Arn": "arn:aws:sts::123423341127:assumed-role/powershell-test-role/botocore-session-1637708010"
}

PS /> $PSVersionTable

Name                           Value
----                           -----
PSVersion                      7.2.0
PSEdition                      Core
GitCommitId                    7.2.0
OS                             Linux 5.4.156-83.273.amzn2.x86_64 #1 SMP Sat Oct 30 12:59:07 UTC 2021
Platform                       Unix
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

PS /> cat /etc/issue
Ubuntu 20.04.3 LTS \n \l

Test info:

Create a new role called powershell-test-role in the account & update the trust to match your EKS cluster for IRSA
powershell-test-role-trust.json.txt

Setup a standard EKS Cluster
Use this set of YAML (update the role ARN in the ServiceAccount
powershell-test.yaml.txt

Connect in over kubectl exec
Run

apt-get update
apt-get -y install ca-certificates unzip curl git gnupg apt-transport-https pkg-config
curl https://packages.microsoft.com/config/ubuntu/20.04/packages-microsoft-prod.deb --output packages.deb
dpkg -i ./packages.db
 apt-get update
 apt-get install powershell
 pwsh -c Install-Module -Name AWSPowerShell.NetCore -Repository PSGallery -Force -ErrorAction Stop
 pwsh
 import-module AWSPowershell.NetCore
 Get-STSCallerIdentity

[ashishdhingra]

Reproducible with output of Get-STSCallerIdentity different from aws sts get-caller-identity.

STEPS:

• Create cluster with Ubuntu 20.04 LTS and with IAM OIDC associated.

eksctl create cluster --name ubuntu-powershell-test --version 1.21 --node-ami-family=Ubuntu2004 --with-oidc

Namespace is kube-system. Use this in subsequent commands and deployment templates.

• Create IAM Service Account:

eksctl create iamserviceaccount --name my-serviceaccount --namespace kube-system --cluster ubuntu-powershell-test --attach-policy-arn arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess --approve

• Use the below template and execute command kubectl apply -f ubuntu-powershell-test.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ubuntu
  labels:
    app: ubuntu
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: ubuntu
  replicas: 1
  template:
    metadata:
      name: ubuntu
      labels:
        app.kubernetes.io/name: ubuntu
    spec:
      serviceAccountName: "my-serviceaccount"
      nodeSelector:
        beta.kubernetes.io/arch: amd64
      containers:
      - name: ubuntu
        image: "ubuntu:20.04"
        imagePullPolicy: Always
        command: ["/bin/sh"]
        args: ["-c", "while true; do echo hello; sleep 10;done"]

• Watch for pod creation in the namespace using command 
kubectl get pods --namespace kube-system -o wide --watch

• Once the Ubuntu pod is created, open interactive session on Ubuntu node using command kubectl exec --stdin --tty ubuntu-56c5bf5cf6-68hdp --namespace kube-system -- /bin/bash

• On the connected Ubuntu pod,

• • Execute the below commands in connected pod to install PowerShell:

apt-get update
apt-get -y install ca-certificates unzip curl git gnupg apt-transport-https pkg-config
curl https://packages.microsoft.com/config/ubuntu/20.04/packages-microsoft-prod.deb --output packages.deb
dpkg -i ./packages.deb
 apt-get update
 apt-get install powershell
 pwsh -c Install-Module -Name AWSPowerShell.NetCore -Repository PSGallery -Force -ErrorAction Stop

• • Execute the below commands in connected Ubuntu pod to install AWS CLI:

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
./aws/install

• • Examine the AWS environment variables using command env | grep AWS:

AWS_ROLE_ARN=arn:aws:iam::<<ACCOUNT-ID>>:role/eksctl-ubuntu-powershell-test-addon-iamservi-Role1-1ENWEJZZEEZNE
AWS_DEFAULT_REGION=us-east-2
AWS_REGION=us-east-2
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token

• • Execute aws sts get-caller-identity in the connected pod and examine the output:

{
    "UserId": "AROASA6NRDFTRA3E5WHQ2:botocore-session-1643830450",
    "Account": "139480602983",
    "Arn": "arn:aws:sts::<<ACCOUNT-ID>>:assumed-role/eksctl-ubuntu-powershell-test-addon-iamservi-Role1-1ENWEJZZEEZNE/botocore-session-1643830450"
}

• • Execute the below commands in the connected pod to launch PowerShell session and get the result of Get-STSCallerIdentity:

pwsh
Import-Module AWSPowershell.NetCore
Get-STSCallerIdentity | select *

It produces the below output:

LoggedAt         : 02/02/2022 11:34:44
Account          : 139480602983
Arn              : arn:aws:sts::<<ACCOUNT-ID>>:assumed-role/eksctl-ubuntu-powershell-test-nod-NodeInstance
                   Role-3H8K5EYL8LDE/i-0357f9a442d1eeba4
UserId           : AROASA6NRDFTWYT5EEQAO:i-0357f9a442d1eeba4
ResponseMetadata : Amazon.Runtime.ResponseMetadata
ContentLength    : 508
HttpStatusCode   : OK

Notice that output of aws sts get-caller-identity is pointing to the value specified by environment variable AWS_ROLE_ARN. Whereas the output of Get-STSCallerIdentity is pointing to node instance role.

[abstrask]

I encountered the same issue and managed to find a workaround:

Import-Module AWSPowershell.NetCore
$AWS_CREDS = Use-STSWebIdentityRole -RoleArn $env:AWS_ROLE_ARN -RoleSessionName $env:HOSTNAME -WebIdentityToken $(gc $env:AWS_WEB_IDENTITY_TOKEN_FILE) -Select 'Credentials'
Set-AWSCredential -Credential $AWS_CREDS
Get-STSCallerIdentity

Haven't refactored my workload yet, but I guess I would need to implement logic to ensure the creds are periodically updated.

[mark-hubers]

I encountered the same issue and managed to find a workaround:

Import-Module AWSPowershell.NetCore
$AWS_CREDS = Use-STSWebIdentityRole -RoleArn $env:AWS_ROLE_ARN -RoleSessionName $env:HOSTNAME -WebIdentityToken $(gc $env:AWS_WEB_IDENTITY_TOKEN_FILE) -Select 'Credentials'
Set-AWSCredential -Credential $AWS_CREDS
Get-STSCallerIdentity

Haven't refactored my workload yet, but I guess I would need to implement logic to ensure the creds are periodically updated.

Thanks so much, abstrask! This worked, but still weird why we have to do this.

I end up adding this to my scripts so I can still run them both in pods or on EC2 instances.

## Fix some weird problem when running inside a AWS EKS pod IAM it not pick up pod role base
if ( $env:AWS_ROLE_ARN -ne $null -and $env:AWS_WEB_IDENTITY_TOKEN_FILE -ne $null ) {
   ## We are running in AWS EKS using POD base IAM so we have to help get the pod role base working
   $AWS_CREDS = Use-STSWebIdentityRole -RoleArn $env:AWS_ROLE_ARN -RoleSessionName $env:HOSTNAME -WebIdentityToken $(gc $env:AWS_WEB_IDENTITY_TOKEN_FILE) -Select 'Credentials'
   Set-AWSCredential -Credential $AWS_CREDS
}