Copy Data from One Mimir Cluster to another

[JustNZ]

Hello,

we currently have two Mimir Clusters and want to reduce the overall Infrastructure Size to only one.

So question now is, if it's possible to copy the bucket data from Cluster1 to the bucket of Cluster2 without any issues

[pstibrany]

Hello,
yes, copying blocks from long-term storage should cause no problems. You may want to disable compactors in source cluster (Cluster1) when you do that, to avoid blocks being changed (new blocks uploaded, old blocks marked for deletion) during the copy.

[JustNZ]

Hey,
alright cool - thanks for the fast answer

[JustNZ]

Hey @pstibrany ,

I did found an "issue" during copy of the blocks from Cluster1 to Cluster2:

1. inside the blocks s3 of Cluster1 is the same BockID as in the blocks s3 for Cluster2 - how to deal with that?
2. after moving a small amount of blocks from Cluster1 to Cluster2, the compactor on Mimir Cluster2 started to throw the following error:
   ts=2024-03-12T13:48:32.164077229Z caller=compactor.go:652 level=error component=compactor msg="failed to compact user blocks" user=test err="compaction: group 0@17241709254077376921-merge--1695859200000-1695945600000: compact blocks [/var/lib/mimir/data/data-compactor/compact/0@17241709254077376921-merge--1695859200000-1695945600000/01HBFDCC69FYA7DY6Q7N05TYXD /var/lib/mimir/data/data-compactor/compact/0@17241709254077376921-merge--1695859200000-1695945600000/01HBFCHQSCV2KY6HSC1PM6FRTM]: populate block: chunk iter: cannot populate chunk 42949673709 from block 01HBFCHQSCV2KY6HSC1PM6FRTM: checksum mismatch expected:3b12049, actual:f421572c"

[pstibrany]

IDs alone indicate it's the same block to me. Same size confirms that. I don't think you need to start from scratch (although it would be good to understand how blocks got duplicated).

To be sure that blocks are correct, you could download them locally and use tsdb-index-health tool from Mimir repository, with -check-chunks option to check chunks as well. (This will print some details about the block, and could also confirm that these blocks are exactly the same).

[JustNZ]

I did the tsdb-index-health on the block which is in both buckets and it does indeed look the same. But that confuses me a lot:

[JustNZ]

Okay I think I now got a plan how to deal with that situation. Thanks a lot so far for your help.

Now my last question - if I copy the blocks from one Mimir Cluster s3 to the other one. How can I tell the Target Cluster to "use" these new blocks. Because as I said I already copied yesterday a lot of data but still can't query that

[JustNZ]

Okay now I can also see the "old" data - don't ask me how and why, but it works :D

Thanks a lot @pstibrany for all your effort to assist me on that - very appreciate that

[pstibrany]

Okay now I can also see the "old" data - don't ask me how and why, but it works :D

There are some delays (compactor needs to find the blocks, and include them into bucket index, then store-gateways and queriers need to reload bucket index), but after a while, blocks should be queryable.