Restrict cluster prometheus rules to metrics of the according cluster

[fpfuetsch]

Closes #554

[itay-grudev]

@fpfuetsch I have an even better solution. I rewrote almost all the alerts so they can be deployed by the operator chart and the rules are rewritten to match against the cluster name.

Here is an example for CNPGClusterInstancesOnSameNode:

count by (namespace, created_by_name, node) (kube_pod_info{created_by_kind="Cluster"}) > 1

or with a cluster label:

count by ({{ .Values.monitoring.prometheusRule.clusterLabel }}, namespace, created_by_name, node) (kube_pod_info{created_by_kind="Cluster"}) > 1

Here the cluster name is:

labels:
  severity: warning
  {{ .Values.monitoring.prometheusRule.clusterLabel }}: $labels.{{ .Values.monitoring.prometheusRule.clusterLabel }}
  namespace: {{ "{{ $labels.namespace }}" }}
  cnpg_cluster: {{ "{{ $labels.created_by_name }}" }}

I already have these. I just need to test them.

[itay-grudev]

And rewrite them in the context of the operator chart.

[fpfuetsch]

@itay-grudev I have two questions regarding your approach:

• Would it be then still possible to exclude certain alerts for certain clusters (like test clusters with only a single instance)?
• Think about a multi tenant k8s environments, with one team providing the operator while multiple other teams are consuming it to create pg cluster. For the consuming teams it would be useful to add custom labels to the alert rules of their clusters (currently the labels are static: namespace, cnpg_cluster, severity) to filter relevant alerts for them, would this be possible to implement when the rules are managed centralized?

I personally like the idea that alert rules for the clusters are part of the cluster deployment while alert rules for the operator are part of the operator deployment.

[itay-grudev]

No. You'll have to implement the ignore logic in your alert routing or escalation software - like Grafana Oncall.

[fpfuetsch]

So you prefer opt-out (ignoring, silencing) instead of opt-in for alerts?

[itay-grudev]

I am not sure, but it would make sense to have a set of alerts shipped with the operator. The only reason I implemented them at the cluster level is because I never figured out how to implement a Cluster Offline Alert without knowledge of the existence of a cluster.

[fpfuetsch]

@itay-grudev What do you think about merging this change until we found a good way to translate the alerts for the operator chart?