Understanding Tenants

[SunSparc]

What is a tenant?
Why should I care about tenants and tenancy?

I looked up a definition:

tenant | ˈtenənt | noun
"a person who occupies land or property rented from a landlord"
(New Oxford American Dictionary)

In regards to Mimir and Prometheus, who/what is considered to be a tenant?

Furthermore, if I left multitenancy_enabled set to false would I ever regret it? Why would I ever enable multitenancy?

[SunSparc]

I found this defintion in the Reference: Grafana Mimir glossary:

Tenant
A tenant is the owner of a set of series written to and queried from Grafana Mimir.
Grafana Mimir isolates series and alerts belonging to different tenants.

That looks like it could mean that any Prometheus server that is sending metrics to Mimir via remote_write could be considered a tenant. Is that accurate? Is that complete? Or could other things be considered tenants in this context?

[charleskorn]

Tenants are used to isolate data and requests from one another - data and requests for tenant A are isolated from tenant B.

Different tenants can have different limits applied, which can be useful for ensuring a single tenant can't have a negative impact on the whole cluster.

Tenants can be defined in whatever way you'd like. For example, you could have different tenants for different applications or teams, or you can just have one big tenant. Tenants are controlled by the X-Scope-OrgID header on requests to Mimir.

There's more information about multi-tenancy in the authentication and authorisation docs.

[SunSparc]

Thank you, I have read the "Grafana Mimir authentication and authorization" document at least a couple of times already. While it was helpful, it did not really give me a good understanding of what tenants actually are. Your explanation was much more clarifying on this topic.

That is great to know that if I used multitenancy I could use the tag on each individual service that is writing to Mimir, or any combination of them. That really opens up the possibilities.

[celesteking]

It's still VERY unclear what a tenant is and how Grafana separates incoming data.
We're moving from a commercial system where there are hostnames/hosts that contain IP services, OS processes, File Systems, Network Interfaces, etc, you get the idea, -- essentially, what a good old SNMP would output.
I don't understand how that maps to Grafana nomenclature.

[pstibrany]

"Tenant" in Grafana Mimir is an isolation unit for data and queries. That is data for each tenant is stored separately, and each tenant can be configured with different settings (eg. data retention, or allowed query lenghts, etc.) Internally in Grafana Cloud each "Hosted Prometheus" instance maps to a separate tenant.

We're moving from a commercial system where there are hostnames/hosts that contain IP services, OS processes, File Systems, Network Interfaces, etc, you get the idea, -- essentially, what a good old SNMP would output.

You can store all that into single tenant.

[celesteking]

You can store all that into single tenant.

Still unclear how to separate the hosts then. In SMTPv3, we'd configure a username and password different for each host, so we know if someone presents valid creds, he is who we assumed him to be. With mimir, I can't seem to understand what's the proper way to distinguish between hosts. Docs say I should use X-Scope-OrgID header to separate tenants, but you say I shouldn't.

With no separation, it's gonna end up in a huge mess, with different hosts stepping over each other. What do I present to the boss, a graph with anon/noname CPU/memory metrics? This is clearly bullshit.

[pstibrany]

Normally each host sends metrics with its own labels [*]. For example metric node_filesystem_free_bytes (from node_exporter) which tells how much free space there is on a filesystem, also has labels which not only tells which filesystem, but also which host reports this. For example:

node_filesystem_free_bytes{cluster="do-nyc1-demo-infra", device="/dev/vda1", fstype="ext4", instance="pool-e1ro5g0nq-rk82j", job="integrations/node_exporter", mountpoint="/"} 75387174912
node_filesystem_free_bytes{cluster="do-nyc1-demo-infra", device="/dev/vda1", fstype="ext4", instance="pool-e1ro5g0nq-rk82o", job="integrations/node_exporter", mountpoint="/"} 73999695872
node_filesystem_free_bytes{cluster="do-nyc1-demo-infra", device="/dev/vda15", fstype="vfat", instance="pool-e1ro5g0nq-rk82j", job="integrations/node_exporter", mountpoint="/boot/efi"} 117618688
node_filesystem_free_bytes{cluster="do-nyc1-demo-infra", device="/dev/vda15", fstype="vfat", instance="pool-e1ro5g0nq-rk82o", job="integrations/node_exporter", mountpoint="/boot/efi"} 117618688
node_filesystem_free_bytes{cluster="do-nyc1-demo-infra", device="shm", fstype="tmpfs", instance="pool-e1ro5g0nq-rk82j", job="integrations/node_exporter", mountpoint="/run/containerd/io.containerd.grpc.v1.cri/sandboxes/14d0b459d506107ec57f345804249ef5553b3fd5ce19e077961c20129b44d187/shm"} 67108864
node_filesystem_free_bytes{cluster="do-nyc1-demo-infra", device="shm", fstype="tmpfs", instance="pool-e1ro5g0nq-rk82j", job="integrations/node_exporter", mountpoint="/run/containerd/io.containerd.grpc.v1.cri/sandboxes/1d8b555781cee6fe3cd5e0f400fc4340e594ce4428319d9603e62b192b4c7c59/shm"} 67108864

If one is then interested in single host only, it's possible to select host-specific values by using the right query, eg. node_filesystem_free_bytes{instance="pool-e1ro5g0nq-rk82o"}.

There are predefined dashboards to dig into common host metrics, like those exported by node_exporter. (example)

In any case, all this data is stored in a single tenant in Mimir.

[*] for correctness, labels like instance or job are not sent by host exporting the metrics, but by agent reading the metrics (eg. by Prometheus or Grafana Alloy).

[vlad-diachenko]

Hey, @celesteking . We can compare mimir tenant with the database in PostgreSQL. just for the analogy...

Let's say you have a single installation of PostgreSQL, but it contains X separate databases, and you might write data from separate sources to the separate databases. So, the data written to database A does not affect database B....

The same for Mimir, you might have a single cluster but with X tenants, and tenant can be anything you want, it can be environments : dev, preprod, prod, qa, for example. So, you can control access to the environments separately, let's say you want to restrict the list of the users who can have access to prod metrics...
Or each tenant can be mapped to your internal teams, like payments-team, reports-team, devops-team, etc... In this case, you can separate the access, and payments-team won't have access to the reports-team and vice-versa...

IMHO, the main idea of the tenant is data isolation and the ability to have separate configurations that do not interfere with other tenants.

[celesteking]

Okay, they're doing partitioning.
Tenant is a partition in this case.

[celesteking]

Chatgpt replied: Grafana Mimir separates metrics from each host by using labels—like instance or host—to identify and distinguish metrics from different hosts, storing and querying them based on these labels.

Now my question is how to authenticate the sending host. Currently, docs suggest installing a reverse proxy that would do the auth check itself. I'm going to use Alloy and Mimir, so how do I tell Alloy to send proper host and how do I check for that host at the proxy, is that a separate HTTP header? Because if it isn't, the proxy would have to inspect the request body which is a HUGE PAIN IN THE ASS.

[celesteking]

CHATGPT says metrics are in the payload. Imagine letting a random customer of ours (owning the host) send random metrics to our collector. It's a security disaster guys.

[pstibrany]

When Alloy is scraping metrics from host, it is automatically adding job and instance labels to scraped metrics. There's typically no authentication for scraping metrics.

Alloy then forwards all scraped metrics into Mimir. Mimir itself doesn't have any authentication mechanism, for that you will need to add some auth proxy.

[pstibrany]

Imagine letting a random customer of ours

You may want to use separate tenant for each of your customer. That would be perfectly valid use of tenants! (It's what we do too)