-
-
Notifications
You must be signed in to change notification settings - Fork 4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
perf(ObjectStoreStorage): Improve (slow) move on same object bucket #47986
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Needs some tests to verify the behavior and prevent regressions.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me. Changes committed :-).
21172bf
to
b11ab3c
Compare
Tests are still missing though. |
I hope, that the object store tests from https://github.com/nextcloud/server/tree/master/tests/lib/Files/ObjectStore will give us a hint, if this change causes harm. |
Sure there shouldn't be any regressions from this change, but we also need a test that ensures the new behavior is present. So please add a test that moves a file on an object storage and ensures this case is triggered. |
a025436
to
a700954
Compare
ddd2296
to
7c1a141
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Awesome! This is a really good improvement 💙
27984de
to
f724be3
Compare
You need to follow the conventional commit format, so the commit message should be something like |
f724be3
to
2639aa0
Compare
2639aa0
to
0ed03aa
Compare
0ed03aa
to
e70075f
Compare
Oh yes. Sorry, I'm still practicing 😅. |
e70075f
to
305dd6b
Compare
No problem, we are here to help :) I just realized though that I made a mistake myself and ideally it would be |
305dd6b
to
f9f69eb
Compare
It seems that pull requests from a fork are not supported. Should I close this one here and create a new one based on https://docs.nextcloud.com/server/latest/developer_manual/getting_started/development_process.html? |
No I think it's fine for this since it only changes backend stuff and your change is covered by tests. |
@sorbaugh can you merge? |
This commit fixes the issue nextcloud#47856. When you upload a file into a group folder and when you use a single S3 bucket as primary storage, the final move operation hangs for a long time. In the background, Nextcloud initiates a copy-delete sequence from the bucket into the bucket, with causes a lot unnecessary overhead. Nextcloud thinks that the file must be imported to another storage and does not recognize that everything is done on the same object bucket. In that case, the import step can be completely skipped, which saves time, network bandwidth and reduces the load on the object storage. The behavior improves a lot with nextcloud#46013. However, there are still some put messages that are being sent to the object storage when you use an object storage as primary storage and upload files into a group folder. Co-authored-by: Kate <26026535+provokateurin@users.noreply.github.com> Signed-off-by: Christoph Fiehe <c.fiehe@eurodata.de>
f9f69eb
to
9597072
Compare
@cfiehe please don't rebase, it is not necessary unless there actually are conflicts. |
Oh, sorry. |
Thanks for your first pull request and welcome to the community! Feel free to keep them coming! If you are looking for issues to tackle then have a look at this selection: https://github.com/nextcloud/server/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22 |
aaand merged! Thanks @cfiehe for your contribution!! <3 |
Hello there, We hope that the review process is going smooth and is helpful for you. We want to ensure your pull request is reviewed to your satisfaction. If you have a moment, our community management team would very much appreciate your feedback on your experience with this PR review process. Your feedback is valuable to us as we continuously strive to improve our community developer experience. Please take a moment to complete our short survey by clicking on the following link: https://cloud.nextcloud.com/apps/forms/s/i9Ago4EQRZ7TWxjfmeEpPkf6 Thank you for contributing to Nextcloud and we hope to hear from you soon! (If you believe you should not receive this message, you can add yourself to the blocklist.) |
Summary
This commit fixes the issue #47856. When you upload a file into a group folder and when you use a single S3 bucket as primary storage, the final move operation hangs for a long time. In the background, Nextcloud initiates a copy-delete sequence from the bucket into the bucket, with causes a lot unnecessary overhead. Nextcloud thinks that the file must be imported to another storage and does not recognize that everything is done on the same object bucket. In that case, the import step can be completely skipped, which saves time, network bandwidth and reduces the load on the object storage.
The behavior improves a lot with #46013. However, after upload there are still some unnecessary put messages that are being sent to the object storage when you use an object storage as primary storage and upload files into a group folder.
Checklist