Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increase gRPC max recv message size in Bazel client #23078

Conversation

clintharrison
Copy link
Contributor

We have seen invocations where the FailureDetail [presumably] is larger than the 4MiB default (we've seen this when we have many CacheNotFoundException exceptions; the list of missing digests is very very large, and the associated stack traces are even larger).

This causes the Bazel client to exit with a misleading error:

Server terminated abruptly (error code: 8, error message: 'Received message larger than max (9620141 vs. 4194304)', log file: '/mnt/tmpfs/BAZEL_OUTPUT/server/jvm.out')

Somewhat confusingly, the server has not actually terminated, but it's a client-side enforced RESOURCE_EXHAUSTED.

I picked 20 MiB pretty arbitrarily, it's just what happened to be large enough for us.

@github-actions github-actions bot added team-Rules-CPP Issues for C++ rules awaiting-review PR is awaiting review from an assigned reviewer labels Jul 23, 2024
@fmeum
Copy link
Collaborator

fmeum commented Jul 24, 2024

@coeuvre for the long digest lists

@tjgq
Copy link
Contributor

tjgq commented Jul 24, 2024

I don't see an issue with increasing the maximum response size, but I think we should (also) improve the error message and stack trace for CacheNotFoundException; reporting every single missing digest, and a separate stack trace per digest, is unlikely to be useful.

@ulrfa
Copy link
Contributor

ulrfa commented Sep 9, 2024

The pull request #23490 reduces the length of BulkTransferException error messages and avoids mentioning all missing digests. @tjgq or @coeuvre, please let me know if you would like me to also try removing stack traces.

@clintharrison
Copy link
Contributor Author

Thanks, that's a much nicer fix I didn't know how to make 😆

@github-actions github-actions bot removed the awaiting-review PR is awaiting review from an assigned reviewer label Sep 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
team-Rules-CPP Issues for C++ rules
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants