feat: rework anchor worker scaling algorithm #17
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
RES-1113 Rework anchor worker scaling algorithm
The current algorithm could spawn a large number of workers because it doesn't take into account how many workers are truly in progress. Each time it loops back to check for the size of the batch queue, it could spawn new workers even if the previously counted batch events had not even started being serviced yet. |
smrz2001
force-pushed
the
feature/res-1113-rework-anchor-worker-scaling-algorithm
branch
from
a6769d6
to
af11655
Compare
smrz2001
deleted the
feature/res-1113-rework-anchor-worker-scaling-algorithm
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Instead of using just the number of batches in flight, this code now uses actual CD manager jobs and states.
This prevents new jobs from being spuriously created if, for any reason, anchor workers could not be started. This could happen if there's a deployment in progress, there's a bug in the CD manager starting anchor workers, there's a bug in the anchor worker that causes it to crash, etc.