how can check 300GB file integrity(checksum) downloaded on network path from s3 payerbucket in c# #3306
Answered
by
ashishdhingra
krishsingh-git
asked this question in
Q&A
how can check 300GB file integrity(checksum) downloaded on network path from s3 payerbucket in c#
#3306
-
|
how can check 300GB file integrity(checksum) downloaded on network path from s3 payerbucket in c# getmetadata for each part failed due to multiple bad request aws s3 bucket |
Beta Was this translation helpful? Give feedback.
Answered by
ashishdhingra
May 3, 2024
Replies: 3 comments 1 reply
-
|
@krishsingh-git Good afternoon. Could you please elaborate your scenario, including a code sample and error you are getting, where you specify Thanks, |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Can we connect on call?i can explain better there
From: Ashish Dhingra ***@***.***>
Sent: Tuesday, April 30, 2024 3:40 AM
To: aws/aws-sdk-net ***@***.***>
Cc: Krishan Singh ***@***.***>; Mention ***@***.***>
Subject: Re: [aws/aws-sdk-net] how can check 300GB file integrity(checksum) downloaded on network path from s3 payerbucket in c# (Discussion #3306)
CAUTION: This email is from an external source. Please don't open any unknown links or attachments.
@krishsingh-git<https://github.com/krishsingh-git> Good afternoon. Could you please elaborate your scenario, including a code sample and error you are getting, where you specify getmetadata for each part failed due to multiple bad request aws s3 bucket? AWS .NET SDK does not support multi-part download as of now. Hence, how are you calling GetObjectMetadata for each part!
Thanks,
Ashish
-
Reply to this email directly, view it on GitHub<#3306 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BADY3SIUMTL6UW5WA4INNN3Y73ALFAVCNFSM6AAAAABG4Y2RYKVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TENRYGA4DC>.
You are receiving this because you were mentioned.Message ID: ***@***.******@***.***>>
This email and the information contained herein is proprietary and confidential and subject to the Amdocs Email Terms of Service, which you may review at https://www.amdocs.com/about/email-terms-of-service <https://www.amdocs.com/about/email-terms-of-service>
|
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Hi Ashis,
I am facing one more issue ,
I am trying to verify hash value of downloaded file from s3 bucket.
But MD5 value is coming different in local,
Can you please look into my code
"sourceUrl":"s3://nprd-cntdel-wh-s3/VUBIQUITY_UK/Source/615685.mxf"
```
using (var outputStream = new MemoryStream())
{
using (MD5 md5 = MD5.Create())
{
using (FileStream fileStream = new FileStream(MD5FileIntegrity.Path.Value, FileMode.Open, FileAccess.Read))
{
byte[] bufferLast = new byte[0];
int i = 1;
while (i <= MD5FileIntegrity.NumberOfParts.Value)
{
Result<GetObjectMetadataResponse> partMetadataResult = await _s3Service.GetObjectMetaDataRetryTokenAsync(s3Location, i, amazonS3, s3ExtendedCopyClient);
long chunkSize = partMetadataResult.Value.ContentLength;
byte[] buffer = new byte[(int)chunkSize];
bufferLast = new byte[(int)chunkSize];
var bytesRead = await fileStream.ReadAsync(buffer, 0, buffer.Length).ConfigureAwait(false);
var bytesReads = fileStream.Read(bufferLast, 0, bufferLast.Length);
Console.WriteLine("bytesRead is " + bytesReads);
md5.TransformBlock(buffer, 0, bufferLast.Length, null, 0);
outputStream.Write(buffer, 0, bytesReads);
i++;
}
md5.TransformFinalBlock(outputStream, 0, outputStream.Length);
}
var myChecksum = BitConverter.ToString(md5.Hash).Replace("-", "").ToUpper();
```
From: Ashish Dhingra ***@***.***>
Sent: Tuesday, April 30, 2024 3:40 AM
To: aws/aws-sdk-net ***@***.***>
Cc: Krishan Singh ***@***.***>; Mention ***@***.***>
Subject: Re: [aws/aws-sdk-net] how can check 300GB file integrity(checksum) downloaded on network path from s3 payerbucket in c# (Discussion #3306)
CAUTION: This email is from an external source. Please don't open any unknown links or attachments.
@krishsingh-git<https://github.com/krishsingh-git> Good afternoon. Could you please elaborate your scenario, including a code sample and error you are getting, where you specify getmetadata for each part failed due to multiple bad request aws s3 bucket? AWS .NET SDK does not support multi-part download as of now. Hence, how are you calling GetObjectMetadata for each part!
Thanks,
Ashish
-
Reply to this email directly, view it on GitHub<#3306 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BADY3SIUMTL6UW5WA4INNN3Y73ALFAVCNFSM6AAAAABG4Y2RYKVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TENRYGA4DC>.
You are receiving this because you were mentioned.Message ID: ***@***.******@***.***>>
This email and the information contained herein is proprietary and confidential and subject to the Amdocs Email Terms of Service, which you may review at https://www.amdocs.com/about/email-terms-of-service <https://www.amdocs.com/about/email-terms-of-service>
|
Beta Was this translation helpful? Give feedback.
1 reply
-
|
@krishsingh-git You could try checking Checking object integrity and New – Additional Checksum Algorithms for Amazon S3. SDK(s) also have support for flexible checksums.
Thanks, |
Beta Was this translation helpful? Give feedback.
Answer selected by
ashishdhingra
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
@krishsingh-git You could try checking Checking object integrity and New – Additional Checksum Algorithms for Amazon S3.
SDK(s) also have support for flexible checksums.
getObjectRequest.ChecksumMode = ChecksumMode.Enabledto trigger the SDK to validate the integrity as the response stream is read.Thanks,
Ashish