Alerts¶
Compliant Kubernetes (CK8S) includes alerts via Alertmanager.
Important
By default, you will get some platform alerts, specifically those originating from the workload cluster. This may benefit you, by giving you improved "situational awareness". Please decide if these alerts are of interest to you or not. Feel free to silence them, as the Compliant Kubernetes administrator will take responsibility for them.
Your focus should be on user alerts or application-level alerts, i.e., alerts under the control and responsibility of the Compliant Kubernetes user. We will focus on user alerts in this document.
Compliance needs¶
Many regulations require you to have an incident management process. Alerts help you discover abnormal application behavior that need attention. This maps to ISO 27001 – Annex A.16: Information Security Incident Management.
Enabling user alerts¶
User alerts are handled by a project called AlertManager, which needs to be enabled by the administrator. Get in touch with the administrator and they will be happy to help.
Configuring user alerts¶
User alerts are configured via the Secret alertmanager-alertmanager
located in the alertmanager
namespace. This configuration file is specified here.
# retrieve the old configuration
kubectl get -n alertmanager secret alertmanager-alertmanager -o jsonpath='{.data.alertmanager\.yaml}' | base64 -d > alertmanager.yaml
# edit alertmanager.yaml as needed
# patch the new configuration
kubectl patch -n alertmanager secret alertmanager-alertmanager -p "'{\"data\":{\"alertmanager.yaml\":\"$(base64 -w 0 < alertmanager.yaml)\"}}'"
Make sure to configure and test a receiver for you alerts, e.g., Slack or OpsGenie.
Note
If you get an access denied error, check with your Compliant Kubernetes administrator.
Accessing user AlertManager¶
If you want to access AlertManager, for example to confirm that its configuration was picked up correctly, proceed as follows:
- Type:
kubectl proxy
. - Open this link in your browser.
Setting up an alert¶
Before setting up an alert, you must create a ServiceMonitor to collect metrics from your application. Then, create a PrometheusRule
following the example below:
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
creationTimestamp: null
labels:
prometheus: example
role: alert-rules
name: prometheus-example-rules
spec:
groups:
- name: ./example.rules
rules:
- alert: ExampleAlert
expr: vector(1)
Running Example¶
The user demo already includes a PrometheusRule, to configure an alert:
{{- if .Values.prometheusRule.enabled -}}
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: {{ include "ck8s-user-demo.fullname" . }}
labels:
{{- include "ck8s-user-demo.labels" . | nindent 4 }}
spec:
groups:
- name: ./example.rules
rules:
- alert: ApplicationIsActuallyUsed
expr: rate(http_request_duration_seconds_count[1m])>1
{{- end }}
The screenshot below gives an example of the application alert, as seen in AlertManager.