Query multiple S3 endpoints

[dili-pk]

Hi,

I am working on setting up Grafana Mimir on a very large infrastructure which is configured with multiple regions(each region has 70 - 100 kubernetes clusters).
We have around 6 regions where we have deployed 6 different mimir clusters and each mimir cluster writing to 6 different S3 endpoints.
Now we are trying to create a query cluster which has only query front-end, querier and query scheduler deployed to query across all 6 clusters.
I was thinking of querying the data from these 6 different S3 buckets or from different Store-gateways. But I am not sure if we can configure multiple S3 endpoints.. This doesn't seem to work and do not find a configuration option..

Any thoughts how this can be achieved? Or is this currently not possible?

[dimitarvdimitrov]

what's the write distribution between those 6 endpoints? My understanding is that each mimir cluster writes to 6 different endpoints. Writing happens by the ingester and compactor. How did you configure them to write to multiple clusters? IIRC this isn't supported by mimir out of the box.

Having said that, it might be easier if you have each mimir cluster store its data in a single endpoint. This way the clusters have disjoint data. You can query across all of the using GEM's federation-frontend component. It should be able to run without a valid GEM license, so you can use it with Mimir.

[dili-pk]

Yeah.. I agree on that.. It is better to have a single S3 endpoint for all the 6 Mimir clusters. However, our infrastructure is too big and we are bit concerned on the latency of reads and writes.
We currently have a setup with Thanos the same way.. We also have a query cluster which connects to all the different store-gateway of other thanos deployments to fetch the metrics and query it globally, we are trying to replicate the same and I am not finding a way with Mimir..

[dimitarvdimitrov]

i don't this it's possible with Mimir. None of the Mimir are multi-bucket aware. This means that even if you set up different store-gateway pools for different endpoints, none of the other Mimir components know how to treat these store-gateway as having different data.

The only approach that I can think of are fully-fledged query paths for each bucket: query-frontend, query-scheduler, querier, store-gateway. And then put a federation-frontend in front of each of the 6 query paths so it interleaves the 6 results into a single result.

[dili-pk]

Thank you for the information..
I am trying to explore the federation-frontend now. However, the document seems to be a bit confusing on where exactly this feature has to be used?
Do we need to enable GEM in Mimir and use this feature or directly in Mimir configuration?
I directly tried using this in our Mimir Config and I was getting unmarshall error for federation.. It doesn't take federation as an option within the config.
Not sure if Iam doing it right. Do you have any deployment config example I can take as a reference?

Mimir.yaml:

target: federation-frontend
    federation:
      proxy_targets:
      - name: cluster-1
        url: <host>/mimir/prometheus
common:
  storage:
    backend: s3
    s3:
      access_key_id: ZXXXXXXXXX
      endpoint: <minio-endpoint>:9000
      insecure: true
      secret_access_key: XXXXXXXXXXXX

Thanks again !

[dimitarvdimitrov]

you would have to deploy a GEM image (grafana/enterprise-metrics) and then configure it with the federation parameters to route queries to all the Mimir clusters. It will have to be deployed as a separate deployment in addition to all the Mimir components/clusters

[vg-meesho]

@dili-pk have you solved this problem? If so change you share the solution. Thanks