[META] Remote Cluster State Issues (Phase - 2)

[linuxpi]

Is your feature request related to a problem? Please describe.
Issues listed under this meta issue are related Remote Cluster State feature for Phase 2. As part of Phase 2, we will add support to backup and restore global metadata to and from remote store.

Describe the solution you'd like
#9143

Tasks

OpenSearch 2.10 Release

• [Remote State] Restore Global Metadata from Remote state after quorum loss recovery. #10525
• [Remote State] Upload Global Metadata to Remote Store #10526
• [Remote State] Download Global Metadata from Remote Store #10527
• [Remote Cluster State] Allow Remote Cluster state writes to remote use a dedicated threadpool #9816
• [Remote State] Update cluster UUID trimming logic for remote state auto restore flow to compare global metadata #10662
• [BUG] [Remote State] Auto Restore during remote cluster state restore can lead to split primaries in Remote Store Domains #10658
• [BUG] [Remote State] Cluster chaining can break if there is consecutive node replace and restart #10841
• [Remote cluster state] Few custom metadata not getting persisted on storage which dont support async write. #10697
• [BUG] [Remote State] Some custom entities of Metadata will not be restored for Repositories not implementing AsyncMultiStreamBlobContainer #10691
• [BUG] Fix exception handling for global metadata upload #10888
• [BUG] [Remote State] NoSuchFileException during stale manifest deletion within a cluster UUID in the upload path #10586
• [BUG] [Remote Store] Post cluster manager recovery from local disk, system_repository setting is always false #10555
• [BUG] [Remote Cluster] Local State recovery should bring up remote indices in GREEN state #10071
• [BUG] [Remote State] During construction of valid chain we skip valid cluster UUIDs if the are previousClusterUUIDs for invalid UUIDs #10911
• Fix logging for the nodes with remote store attributes #10809
• [BUG] Remote cluster state download/restore is failing for Data stream #10773
• [Remote cluster state] Make timeout as dynamic settings. #10688
• [Remote cluster state] Emit metrics for timeout for metadata transfer to remote store #10687
• [Remote cluster state] Optimise updating global metadata and index metadata #10645
• [Remote Cluster State] Ensure proper closure of IndexInput created in ChecksumBlobStoreFormat #10978
• [BUG] [Remote store] Throttling from remote store shouldn't cause leader stepping down and re-elections #10683
• [Optimization] [Remote state] With every election full state shouldn't get uploaded to remote #10682
• [BUG] [Remote State] Garbage collect dangling metadata files in remote state repo #10686
• [BUG] AsyncTransferManager eatingup exception #10890
• [Remote State] Make manifest files cleanup less aggressive #10990
• [BUG] [Remote State] Add cross chain trimming to handle mutliple valid chains having same state #10913
• [BUG] [Remote State] Auto restoring decommission attributes after QL can lead to election loops #10905
• <adjust remote retries to have a shorter back-off/more attempts as compared to translog/segments>

Additional context
Previous Release - #9344

[kiranprakash154]

Hi, are we on track for this to be released in 2.12 ?

[shwetathareja]

@soosinha can you update this Meta issue with latest status?

[soosinha]

The changes planned for 2.12 have been released. The major changes are upload of global metadata on cluster state publication and download of global metadata during cluster-manager node bootstrap. The timeout of metadata upload has been added as dynamic settings. Cluster UUID chaining logic has been modified according to global metadata. Cluster state upload stats having added under discovery metric in node stats API. Cluster state uploads are now being performed from a separate threadpool.
Resolving this meta issue now. The pending issues will be tracked individually