ListObjectsAsync Throws AmazonS3Exception: The specified key does not exist.

[StrangeWill]

When we call client.ListObjectsAsync(bucketName) we get the error AmazonS3Exception: The specified key does not exist., it's a bit weird for us on a few things:

• We have GetObject/ListBucjet/PutObject/DeleteObject on the bucket
• We use the same configuration for reading/writing objects to the bucket, works fine
• We've handed it access keys for an account granted AmazonS3FullAccess

Am I missing anything here?

[ashishdhingra]

@StrangeWill The AmazonS3Client.ListObjectsAsync() is auto-generated and simply invokes ListObjects operation. So it appears to be more of permissions issue. could you please check the following:

• Are you using the right account for making the S3 call? You can verify this by using AWS CLI command aws sts get-caller-identity?
• Could you share you policy (with sensitive information removed)? Does it have Resource condition which might not be setup properly (refer StackOverflow post)?

Thanks,
Ashish

[StrangeWill]

Policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ListObjects",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::[id]:user/[app user]",
                    "arn:aws:iam::[id]:user/[my account]",
                    "arn:aws:iam::[id]:user/[another staff account]"
                ]
            },
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject"
            ],
            "Resource": [
				"arn:aws:s3:::[bucket name]",
				"arn:aws:s3:::[bucket name]/*"
			]
        },
        {
            "Sid": "ListBucket",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::[id]:user/[app user]",
                    "arn:aws:iam::[id]:user/[my account]",
                    "arn:aws:iam::[id]:user/[another staff account]"
                ]
            },
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::[bucket name]"
        }
    ]
}

We also tried setting "Everyone" access to list the bucket (nothing sensitive in it, but it's just for a test anyway), and still we're getting the error. I've tried using access keys from both the arn:aws:iam::[id]:user/[app user]" and my own arn:aws:iam::[id]:user/[my account] account.

No problem with our Minio systems, only Amazon's S3 stuff.

[ashishdhingra]

@StrangeWill Unfortunately, I'm unable to reproduce the issue. Below are the steps taken:

• Created a new user named testuser that has programmatic access (with access and secret keys), with NO IAM policy attached.
• In AWS Console, for the existing bucket, added the bucket policy in the Permissions tab:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ListBucket",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<<account-id>>:user/testuser"
            },
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::<<bucket-name>>"
        },
        {
            "Sid": "ListObjects",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<<account-id>>:user/testuser"
            },
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject"
            ],
            "Resource": "arn:aws:s3:::<<bucket-name>>/*"
        }
    ]
}

Notice that I did not add resource arn:aws:s3:::<<bucket-name>> for ListObjects statement since that would be incorrect.

• Developed the below .NET 6 console application:

using Amazon;
using Amazon.S3;
using Amazon.S3.Model;

string accessKey = "<<access-key>>";
string secretKey = "<<secret-key>>";
string bucketName = "<<bucket-name>>";

var client = new AmazonS3Client(accessKey, secretKey, RegionEndpoint.USEast2); // set the region endpoint per your needs.
var response = await client.ListObjectsAsync(new ListObjectsRequest() {
    BucketName = bucketName
});

Console.WriteLine($"Found {response.S3Objects.Count} objects.");

The above program executes without any issues.

In fact, if you only want a user to be able to list bucket contents, then the following bucket policy is sufficient:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ListBucket",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<<account-id>>:user/testuser"
            },
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::<<bucket-name>>"
        }
    ]
}

Not sure if for the specified IAM users, you have permissions (policies) attached to them which prohibits the S3 bucket operation(s).

Thanks,
Ashish

[StrangeWill]

OK, since we're bouncing between AWS and other S3 systems, found the underlying issue: ForcePathStyle = true -- doing this on AWS while it works for everything else we do (uploading/downloading/etc.), will cause an error while listing objects.

We actually have code to disable this when a ServiceURL is not provided, but one was provided for our AWS endpoints. I'll be adjusting that -- thanks for the help on this, and this can be a chunk of knowledge in case anyone else runs into this.

Not sure if this is undesired behavior, but at least I have a workaround.

[github-actions[bot]]

Hello! Reopening this discussion to make it searchable.