Filter by cluster label then data source¶
- Status: accepted
- Deciders: arch meeting
- Date: 2021-01-27
Technical Story: https://github.com/elastisys/compliantkubernetes-apps/issues/742
Context and Problem Statement¶
Compliant Kubernetes allows multiple workload clusters to be connected to a single service cluster. This allows the metrics of multiple workload clusters to be inspected via the same dashboards.
How should we organise metrics to allow users and admins to select for which clusters they want to see metrics?
- We want to be able to see metrics for a single cluster, for multiple cluster, and even for all clusters.
- We want to be able to reuse upstream dashboards, and some are missing filters for the
- We want to stay flexible.
- Use only the
clusterlabel and expose a single data source.
- Expose multiple data sources and ignore the
- Filter primarily by
clusterlabel, but allow filtering by data source.
- Filter primarily by data source, but allow filtering by
"Filter primarily by
cluster label, but allow filtering by data source",
because it fulfills the all decision drivers with little complexity.
Prom-label-enforcer can be used to create multiple data sources from a single data store, discriminating by
cluster label. To simplify Thanos configuration, we can also discriminate based on
tenant_id, which will always contain the same value as
In general, we will aim to fix dashboards missing the
cluster variable upstream. However, by also providing filtering based on data source, we facilitate our users to reuse their dashboards, which might not be cluster-aware.
- We support both dashboards with
clusterfilter and without
- We can enforce metrics multi-tenancy, i.e., map Grafana users/orgs to datasources, to filter some metrics out.
- [Minor] We need to configure data sources in
- For example, if we forget to add the name of a workload cluster, the data source will be missing, but filtering based on
clusterlabel is still possible.
- [Minor] Label enforcer uses a bit of resources.
- However, we already saved a lot by migrating from InfluxDB to Thanos, so we can afford go back a bit.