Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Checksums not Calculated on Azure for files > 32MB #1805

Open
cancan101 opened this issue Aug 3, 2024 · 0 comments
Open

Checksums not Calculated on Azure for files > 32MB #1805

cancan101 opened this issue Aug 3, 2024 · 0 comments

Comments

@cancan101
Copy link

cancan101 commented Aug 3, 2024

Bug Report

Q A
Flysystem Version 3.28.0
Adapter Name flysystem-azure-blob-storage
Adapter version 3.28.0

Summary

Azure will calculate md5 checksums only in the case where a file is uploaded in a single chunk. (See SO post for more info).

The chunk size defaults to 32MB but can be raised up to 265MB. Anything over that would require local calculation of the MD5 hash. Alternatively, Azure offers an ETag which seems to be calculated for larger files, though this value doesn't look like it corresponds to a content hash.

Not having Checksums reliably calculated causes issues with Pimcore, which assumes the checksum method won't error out. I filed downstream ticket: pimcore/pimcore#17438.

How to reproduce

Create a file larger than 32MB (e.g. with mkfile -n 44m ~/Downloads/44mb) and then call checksum on that file.

Suggestion

Expose an option to set singleBlobUploadThresholdInBytes on the client.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant