branch-4.1: [fix](cloud) Drop CloudReplica route entries of dropped compute groups #66984 - #67122
Open
github-actions[bot] wants to merge 1 commit into
Open
branch-4.1: [fix](cloud) Drop CloudReplica route entries of dropped compute groups #66984#67122github-actions[bot] wants to merge 1 commit into
github-actions[bot] wants to merge 1 commit into
Conversation
#66984) `CloudReplica.primaryClusterToBackend` (`be`, previously `bes`) is persisted per replica, but nothing ever removes an entry: `CloudSystemInfoService .dropCluster()` only touches the system-info maps, `CloudTabletRebalancer .loopCloudReplica()` only walks the compute groups that currently exist, and `unprotectUpdateCloudReplica()` only puts. So every created/dropped compute group leaves one key behind on every replica, forever. On an instance that had repeatedly added and removed compute groups this grew to 392 route keys per replica against 3 live compute groups: 45.5M entries over 167k replicas, 1.78 GB of the 2.07 GB image (85.9%) and ~8.6 GB of live heap. Metadata checkpoint keeps the online catalog and the checkpoint catalog resident at the same time, which no longer fit in the heap, so checkpoint OOMed on every leader in turn and the image never advanced. Drop an entry once its backend id no longer resolves. Such an entry is never routed to: `getBackendIdImpl()` already resolves the id, gets null and falls back to `hashReplicaToBe()`, so it only costs memory and image bytes. No edit log op is written for the removal, and none is needed: `Checkpoint` is a `MasterDaemon`, and its Env cleans the catalog it loads before saving, so the image is clean no matter what any serving Env holds. The rebalancer sweep is master-only, so a follower keeps its own stale entries in heap until it restarts or is promoted, at which point one rebalancer round clears them. That is bounded, never reaches persisted state, and is the tradeoff for not adding an edit log op that would block downgrades. A dead primary is kept while its compute group still has a live secondary. With `enable_immediate_be_assign=false` -- the default -- primary pointing at an unavailable BE while the secondary holds the rehashed live one is the normal failover state, and `FrontendServiceImpl.getTabletReplicaInfos()` reaches secondaries through `getPrimaryComputeGroupIds()`, so dropping the primary key would hide a live secondary BE from the lazy fetch peer cache candidates. The secondary map is cleaned first, so a dead pair is still fully removed. Two call sites: - `gsonPostProcess()`, outside the `bes` migration branch, so both formats are covered. The backends module is loaded before db/recycleBin, and the checkpoint thread resolves `Env.getCurrentEnv()` to its own Env, so the backend set read there belongs to the image being loaded. This shrinks the load peak and makes existing oversized images converge. - The existing per-replica callback in `completeRouteInfo()`, so a running leader converges too instead of waiting for a restart. It is pinned to one compute group per round because `loopCloudReplica()` invokes the callback once per (replica, compute group) while holding `table.readLock()`. `secondaryClusterToBackends` is not persisted, but leaks the same way in heap, so it is cleaned by the same predicate. Guarded by `enable_cloud_replica_stale_route_clean` (default true). Measured on the metadata of the affected instance. Same build and same `image.22568053` (2,096,733,909 B) for both runs, only the config flipped; the FE was started with `enable_check_compatibility_mode` and `checkpoint_after_check_compatibility`, which loads the image, replays the journal and dumps a new one. | | clean=false | clean=true | | --- | --- | --- | | dumped image | 1,961,181,357 B | 272,087,919 B (-86.1%) | | route entries | 46,004,776 | 353,422 (-99.2%) | | distinct route keys | 395 | 2 | | stored format | `bes` x 176,941 | `be` x 176,926 | | live set after GC | 8.85 GB | 2.73 GB (-69%) | | heap expanded to | 31 GB | stayed at the 8 GB Xms | | dump duration | 72.6 s | 6.7 s | The two surviving route keys are exactly the two compute groups whose backends are present in the image's own backend module -- `cluster_id_1` -> 1760610536752 and `cluster_id_2` -> 1765176237562, both alive and NORMAL -- so no live route was dropped. Feeding the 272 MB result back in with `-Xmx8g` loads, replays and re-dumps it in 20s with a 0.51 GB live set and a stable output size, against the 19.2 GiB heap that could not complete a checkpoint before.
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
|
run buildall |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-picked from #66984