ByteRange is not working in IAmzonS3.GetObjectAsync

[ElectricVampire]

Describe the bug

I am processing a very huge file from S3 hence want to process it by byte chunks.
I am passing the ByteRange parameter in GetObjectAsync while downloading the object from S3 expecting only the bytes asked ByteRange param (which should be 1000 to ContentLength) but I am getting "An error occurred (SignatureDoesNotMatch) when calling the GetObject operation: The request signature we calculated does not match the signature you provided. Check your key and signing method."

var downloadRequest = new GetObjectRequest()
            {
                Key = "data.txt",
                BucketName = "app-test-bucket",
                ByteRange = new ByteRange("bytes=1000-")
            };
 GetObjectResponse s3Response = await _clientAmazonS3.GetObjectAsync(downloadRequest);

Here I would expect content s3Response.ContentLength to be set to ContentLength-1000 as I only asked for "1000- bytes" but getting error.

Debug logs :

AsyncCall = True; CanonicalRequest = GET\n/data.txt\n\nhost:app-test-bucket.s3.amazonaws.com\nrange:bytes=1000-\nuser-agent:aws-sdk-dotnet-coreclr/3.7.101.15 aws-sdk-dotnet-core/3.7.100.15 .NET_Core/6.0.10 OS/Microsoft_Windows_10.0.14393 ClientAsync\nx-amz-content-sha256:XXXXX\nx-amz-date:20230117T085013Z\nx-amz-security-token:XXXXX\n\nhost;range;user-agent;x-amz-content-sha256;x-amz-date;x-amz-security-token\XXXXX; StringToSign = AWS4-HMAC-SHA256\n20230117T085013Z\n20230117/us-east-1/s3/aws4_request\nXXXXXXX; ServiceName = AmazonS3; ServiceEndpoint = https://app-test-bucket.s3.amazonaws.com/; MethodName = GetObjectRequest; Exception = Amazon.Runtime.Internal.HttpErrorResponseException: Exception of type 'Amazon.Runtime.Internal.HttpErrorResponseException' was thrown.\r\n at Amazon.Runtime.HttpWebRequestMessage.GetResponseAsync(CancellationToken cancellationToken)\r\n at Amazon.Runtime.Internal.HttpHandler`1.InvokeAsync[T](IExecutionContext executionContext)\r\n at Amazon.Runtime.Internal.RedirectHandler.InvokeAsync[T](IExecutionContext executionContext)\r\n at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext)\r\n at Amazon.S3.Internal.AmazonS3ResponseHandler.InvokeAsync[T](IExecutionContext executionContext)\r\n at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext); StatusCode = Forbidden; AmzId2 = XXXXXX; AWSRequestID = WBHD681WVR5ZNCAF; AWSErrorCode = SignatureDoesNotMatch; CredentialsRequestTime = 00:00:00.0030147; RequestSigningTime = 00:00:00.0384433; HttpRequestTime = 00:00:02.0419348; ClientExecuteTime = 00:00:02.405368

Expected Behavior

Here I would expect content s3Response.ContentLength to be set to ContentLength-1000 (all the bytes after 1000th byte) as I only asked for 1000- bytes but getting error.

Current Behavior

GetObjectAsync is not considering ByteRange input param and returning error.

Reproduction Steps

Send the request and you will see an error.

var downloadRequest = new GetObjectRequest()
            {
                Key = "data.txt",
                BucketName = "app-test-bucket",
                ByteRange = new ByteRange("bytes=1000-")
            };
GetObjectResponse s3Response = await _clientAmazonS3.GetObjectAsync(downloadRequest);

Possible Solution

No response

Additional Information/Context

No response

AWS .NET SDK and/or Package version used

AWSSDK.S3.3.7.101.15

Targeted .NET Platform

.NET 6

Operating System and version

Windows 10

[ashishdhingra]

@ElectricVampire Thanks for opening the issue. Upon analysis, it appears that if you don't specify the upper bound, it defaults from the starting byte to the end byte of the object. For instance, the below code:

using Amazon.S3;
using Amazon.S3.Model;

var _clientAmazonS3 = new AmazonS3Client();

var downloadRequest = new GetObjectRequest()
{
    Key = "<<some-key>>",
    BucketName = "<<some-bucket>>",
    ByteRange = new ByteRange("bytes=1000-")
};
GetObjectResponse s3Response = await _clientAmazonS3.GetObjectAsync(downloadRequest);

Console.WriteLine($"Content Length: {s3Response.ContentLength}, Content Range: {s3Response.ContentRange}");

returns

• s3Response.ContentRange as bytes 1000-4945176/4945177
• s3Response.ContentLength as 4944177.

If we change the downloadRequest to:

var downloadRequest = new GetObjectRequest()
{
    Key = "<<some-key>>",
    BucketName = "<<some-bucket>>",
    ByteRange = new ByteRange(0, 1000)
};

it returns:

• s3Response.ContentRange as 0-1000/4945177
• s3Response.ContentLength as 1001.

This is correct. So in your case, you are requesting starting with byte 1000 with open upper limit.

Also, I'm not getting any signature mismatch error while executing the above code.

Hope this provides explanation for your use case.

Thanks,
Ashish

[ElectricVampire]

Code is same. I am even able to reproduce it with aws cli command. So it must be something with data in file.
So is there any restriction on which byte location I can start a read from in S3 stream?

[ashishdhingra]

@ElectricVampire I believe GetObjectAsync() would return the number of bytes represented by s3Response.ContentRange for a file, which could be used to write to a file.

Here's the sample working code (console application), which downloads an S3 object with 1000 bytes at a time using ByteRange property (code is only for demonstration purposes only):

using Amazon.S3;
using Amazon.S3.Model;

var _clientAmazonS3 = new AmazonS3Client();
string downloadPath = "/tmp/temp.csv";
string bucketName = "<<some-bucket-name>>";
string key = "<<some-csv-file-key>>";

int numberOfBytesToRead = 1000;
int startByte = 0;
int endByte = numberOfBytesToRead - 1;

using (FileStream outputFileStream = File.OpenWrite(downloadPath))
{
    do
    {
        ByteRange byteRange = new ByteRange(startByte, endByte);
        var downloadRequest = new GetObjectRequest()
        {
            Key = key,
            BucketName = bucketName,
            ByteRange = byteRange
        };
        GetObjectResponse s3Response = await _clientAmazonS3.GetObjectAsync(downloadRequest);

        byte[] buffer = new byte[numberOfBytesToRead];
        await s3Response.ResponseStream.ReadAsync(buffer, 0, (int) s3Response.ContentLength);
        await outputFileStream.WriteAsync(buffer, 0, (int) s3Response.ContentLength);

        int fileSize = int.Parse(s3Response.ContentRange.Split('/')[1]);

        startByte += numberOfBytesToRead;
        endByte += numberOfBytesToRead;
        if (startByte >= fileSize)
        {
            break;
        }
    }
    while (true);
}

string downloadedContent = await File.ReadAllTextAsync(downloadPath);
Console.WriteLine(downloadedContent);

[ElectricVampire]

@ashishdhingra Tried your code - getting signature mismatch.When we process the large csv file - we want to read it line by line - so I can transform the data of each row and send it to SQS.

So below is how I was trying to do it. I am using -{bytePos} as this is the only thing working for me. Form my testing I think i can't start read from any arbitery bytePos. It gives me signatureError. My new line start from 1296 bytePos and if I try to read it from that bytePos I get signature Error but If I go 3 bytePos behind (2 bytes for CRLF and One in Prev Line) - No signature error.
My lambda will take bytePos as input and it will be invoked via S3Event. Initial invoke will have bytePos set to 0 hence Process will be invoked with 0 pos. After that Process will be invoked with bytePos till where read is completed.

This logic is flawed and doesn't work on large files.
The reason is -

1. I am not able to figure out what is the bytePos for new Line in my file
2. Even if I am able to figure out using some reflection - I can't start read from that Pos - It gives me signature error.

Seed Call: Process(0)

        public async Task<long> Process(long bytePos)
        {
            var downloadRequest = new GetObjectRequest()
            {
                Key = "data.txt",
                BucketName = "app-test-bucket",

            };
           
            // Get the entire stream in first call and read it as much as we can in lambda time
           //  If bytePos is set then get bytes after bytePos
            if (bytePos != 0)
            {
                downloadRequest.ByteRange = new ByteRange($"bytes=-{bytePos}");
            }
            GetObjectResponse s3Response = await _clientAmazonS3.GetObjectAsync(downloadRequest);
            using var sr = new StreamReader(s3Response.ResponseStream);
            string line = String.Empty;
            long totalLen = s3Response.ContentLength;
            long readBytes = 0;
           
           // For some reason we can't start the read from exact first character of line.
           // Hence while returning the bytePos we are going on prevLine for few characters which should be ignored
          // and start a read from new line
            if (bytePos != 0)
            {
                var ignoreLine = await sr.ReadLineAsync();
            }
            Stopwatch timer = new Stopwatch();
            timer.Start();
            while (timer.Elapsed.TotalSeconds < lambdaExecutionTime - 100)
            {
                line = await sr.ReadLineAsync();
                if (line != null)
                {
                    Console.WriteLine(line);
                    readBytes += Encoding.UTF8.GetByteCount(line);
                }
                else
                {
                   // File process is done, we should not invoke this lambda again
                    return -1;
                }
            }
           // Adding 3 bytes - 2 bytes for CRLF and on more to go to previous line as we can't read from first character of line
            var result = totalLen - readBytes + 3;
            sr.Close();
            s3Response.Dispose();
           // Invoke lambda with bytePos set to {result}
            return result;
        }

[ashishdhingra]

This logic is flawed and doesn't work on large files.

@ElectricVampire Are you referring to logic shared earlier in #2523 (reply in thread)? That logic was provided just for guidance and just reads 1000 bytes at a time as part of look to write contents to a file. Your file might contain Windows specific BOM and CRLF characters, and you might want to handle those characters as part of your logic.

Thanks,
Ashish

[ElectricVampire]

@ashishdhingra Nope, I was referring to the code which I provided.

[github-actions[bot]]

Hello! Reopening this discussion to make it searchable.