Skip to main content

Prometheus

Prometheus is a component of NKE that allows you to monitor your applications.

Availability​

Prometheus is available as an optional service for NKE. It can be deployed on an existing NKE cluster using Cockpit.

Usage​

Please see the following sections for an explanation on how to use Prometheus.

General information about the setup​

When Prometheus is ordered, a new Prometheus instance with two replicas will be deployed in your NKE cluster in the nine-system namespace. The pods will run on the control-plane nodes, leaving your node pools fully available for your applications.

Additionally, a new Grafana datasource will be created and automatically registered in your Grafana instance (if you have one deployed).

The Prometheus instance is based on the prometheus-operator project. Therefore, you can use the following resources to create scraping configurations and recording/alerting rules:

  • ServiceMonitors
  • PodMonitors
  • PrometheusRules

It is possible to run multiple Prometheus instances in your cluster if needed.

Exporters and Metrics​

Also, Prometheus comes with some pre-configured metrics exporters:

  • CertManager
  • IngressNginx
  • NodeExporter
  • Kubelet
  • Kubelet cAdvisor
  • KubeStateMetrics
  • NineControllers
  • Velero

You will need to tell us which of these exporters you want to enable. In the future, you will be able to enable them yourself in Cockpit.

Note that enabling all metrics of an exporter will increase the resources required for Prometheus to run. To solve this, you can also limit the amount of metrics to track by explicitly giving us a list of the wanted metrics.

To summarise, we recommend the following workflow:

  • Tell us which exporters to enable and we will enable all metrics for you of said exporter
  • Create your dashboards/rules
  • See which metrics you need and tell us. We will limit the scrape configuration to only scrape your needed metrics.

API preparation steps​

In some of the following paragraphs you might need to use nctl and kubectl to work with the nine Prometheus API resource. Before doing that, please make sure you are logged in to the API and selected a Prometheus instance to work on. After a successful login you will be able to use the nineapis.ch kubeconfig context to execute kubectl commands against the API.

Login to the API​

  1. Make sure that you have kubectl and nctl installed.

  2. Authenticate with our API using nctl:

    nctl auth login

  3. Now you can use kubectl commands

    kubectl --context nineapis.ch get prometheus.observability.nine.ch

Select a Prometheus resource​

All API commands will need a Prometheus resource name and project they can be applied to. We will export the name and project via environment variables for easier access in the examples.

  1. List all your Prometheus instances and select one to work with.

    $ nctl get all --kinds=Prometheus -A

    PROJECT    NAME                  KIND          GROUP
    acme       example               Prometheus    observability.nine.ch
    acme-prod  sample                Prometheus    observability.nine.ch

  2. Export the name and project of the desired Prometheus instance via environment variables.

    export PROMNAME=example PROMPROJECT=acme

Accessing metrics​

There are different ways to view the collected metrics of your Prometheus instance which are explained in the following sections.

Grafana​

This is our recommended way of viewing the metrics of your Prometheus instance(s). Just create a Grafana instance and all your created Prometheus instances will automatically be configured as data sources in it.

Prometheus Web-UI​

Every Prometheus instance provides a web UI which is accessible from outside of the cluster and secured by basic authentication. You will need to gather the connection details before accessing the web UI.

  1. Make sure to execute the API preparation steps.

  2. Check if a connection secret reference was already set for your instance. A referenced connection secret will expose the URL, the basic auth username and the basic auth password.

    kubectl get prometheus.observability.nine.ch $PROMNAME \
            --context nineapis.ch \
            -n $PROMPROJECT \
            -o template --template={{.spec.writeConnectionSecretToRef}}

    If you see an output like <no value>, you will need to set a connection secret reference. Otherwise you can just retrieve the URL and credentials as described further down.

  3. If you need to set a connection secret reference, you can use kubectl for this. In the following example, we will use a secret 'my-prometheus-connection-details' for this, but you can choose your own name. Please make sure that the chosen name does not exist yet (you can use kubectl --context=nineapis.ch get secret -n $PROMPROJECT to check).

    kubectl patch prometheus.observability.nine.ch $PROMNAME \
            --context nineapis.ch \
            -n $PROMPROJECT \
            --type=merge -p '{"spec":{"writeConnectionSecretToRef":{"name":"my-prometheus-connection-details","namespace":"'$PROMPROJECT'"}}}'

  4. Once the connection secret reference was set, we can gather the connection details from the referenced secret. Please make sure to use the name of your chosen connection secret reference.

    kubectl get secret my-prometheus-connection-details \
            --context nineapis.ch \
            -n $PROMPROJECT \
            -o jsonpath='{.data.url}' | base64 --decode
    kubectl get secret my-prometheus-connection-details \
            --context nineapis.ch \
            -n $PROMPROJECT \
            -o jsonpath='{.data.basicAuthUsername}' | base64 --decode
    kubectl get secret my-prometheus-connection-details \
            --context nineapis.ch \
            -n $PROMPROJECT \
            -o jsonpath='{.data.basicAuthPassword}' | base64 --decode

    You can then access the Prometheus Web UI in a browser.

In-cluster access to Prometheus​

Enable In-cluster access​

If you want to access the Prometheus instance from pods running within your NKE cluster, you will need to enable and configure internal cluster access first. As the internal cluster access will not use any authentication, it is disabled by default. One use case for in-cluster access is the operation of a self-managed Grafana instance for example.

  1. Make sure to execute the API preparation steps.

  2. Enable and configure internal access to Prometheus. You can check first if the internal access was already enabled.

    $ kubectl get prometheus.observability.nine.ch $PROMNAME \
            --context nineapis.ch \
            -n $PROMPROJECT \
            -o jsonpath='{.spec.forProvider.access.internal}'

    {"enabled":false,"namespaceSelector":{},"podSelector":{}}

    As you can see in aboves output, internal access was not yet enabled ("enabled" is set to false).

  3. Enabling internal access will, by default, allow every pod of the NKE cluster to connect to Prometheus. You can restrict which pods are allowed to connect via the "namespaceSelector" and "podSelector" fields. They contain label key-value pairs which select the permitted namespace(s) and/or pods based on the labels set on these namespaces/pods. An empty "namespaceSelector" selects pods from all namespaces. An empty "podSelector" field selects all pods from the namespaces which have been selected by the "namespaceSelector" field. Further information for selectors and labels can be found in the official Kubernetes documentation. For example, to allow access from all pods running in a namespace "grafana" you can make use of the special Kubernetes label kubernetes.io/metadata.name which gets automatically attached to every Kubernetes namespace and contains the name of the namespace as a value.

    Here are some examples for enabling the internal access.

    # just enable access for all pods
    kubectl patch prometheus.observability.nine.ch $PROMNAME \
            --context nineapis.ch \
            -n $PROMPROJECT \
            --type=merge -p '{"spec":{"forProvider":{"access":{"internal":{"enabled":true}}}}}'
    # enable access for all pods in the namespace 'grafana'
    kubectl patch prometheus.observability.nine.ch $PROMNAME \
            --context nineapis.ch \
            -n $PROMPROJECT \
            --type=merge -p '{"spec":{"forProvider":{"access":{"internal":{"enabled":true,"namespaceSelector":{"matchLabels":{"kubernetes.io/metadata.name":"grafana"}}}}}}}'
    # enable access for all pods which have a 'app:grafana' label in the namespace 'grafana'
    kubectl patch prometheus.observability.nine.ch $PROMNAME \
            --context nineapis.ch \
            -n $PROMPROJECT \
            --type=merge -p '{"spec":{"forProvider":{"access":{"internal":{"enabled":true,"namespaceSelector":{"matchLabels":{"kubernetes.io/metadata.name":"grafana"}},"podSelector":{"matchLabels":{"app":"grafana"}}}}}}}'

  4. Once you enabled the internal access, you can find the internal URL to access Prometheus in the status of the Prometheus resource itself.

    $ kubectl get prometheus.observability.nine.ch $PROMNAME \
            --context nineapis.ch \
            -n $PROMPROJECT \
            -o jsonpath='{.status.atProvider.internalURL}'

    http://prometheus-internal-3343791.nine-system.svc.cluster.local:9090

    You can then use this URL to access Prometheus from the permitted pods in your cluster.

Disable in-cluster access to Prometheus​

In-cluster access can be disabled by using kubectl.

  1. Make sure to execute the API preparation steps.

  2. Disable access to Prometheus.

    kubectl patch prometheus.observability.nine.ch $PROMNAME \
            --context nineapis.ch \
            -n $PROMPROJECT \
            --type=merge -p '{"spec":{"forProvider":{"access":{"internal":{"enabled":false}}}}}'
Disable external access to Prometheus​

You might have reasons to switch off the external access to the Prometheus web UI. For example, if you run a self managed Grafana instance in your cluster, you will not access Prometheus from outside of the cluster. In these cases you can disable access to Prometheus from external sources. Please note, that nine managed Grafana instances will also not have any access to your Prometheus instance once external access was disabled.

  1. Make sure to execute the API preparation steps.

  2. Disable external access to Prometheus.

    kubectl patch prometheus.observability.nine.ch $PROMNAME \
            --context nineapis.ch \
            -n $PROMPROJECT \
            --type=merge -p '{"spec":{"forProvider":{"access":{"noExternalAccess":true}}}}'

Instrumenting your application​

Before Prometheus can scrape metrics from your application, you will need to instrument your application to export metrics in a special given format. You can find information about how to do this in the official Prometheus documentation.

Adding application metrics to Prometheus​

Once your application supports metrics, you can use ServiceMonitors or PodMonitors to let Prometheus scrape your application's metrics.

ServiceMonitors will scrape all pods which are targeted by one or more services. This resource needs to be used in most of the cases. You need to define a label selector in the ServiceMonitor which will be used to find all the wanted services. The ServiceMonitor should be created in the same namespace as the service(s) it selects. Next to the label selector your ServiceMonitor should also have the label prometheus.nine.ch/<your prometheus name>: scrape set with the name of your Prometheus instance. Consider the following example ServiceMonitor and Service definition:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-app
  namespace: my-app
  labels:
    prometheus.nine.ch/myprom01: scrape
spec:
  selector:
    matchLabels:
      app: my-app
  endpoints:
    - port: web
kind: Service
apiVersion: v1
metadata:
  name: my-app-service
  namespace: my-app
  labels:
    app: my-app
spec:
  selector:
    application: example-app
  ports:
    - name: web
      port: 8080

The given ServiceMonitor definition will select the service "my-app-service" because the label "app: my-app" exists on that service. Prometheus will then search for all pods which are targeted by this service and starts to scrape them for metrics on port 8080 (the ServiceMonitor defines the port in the endpoints field).

PodMonitors will scrape all pods which are selected by the given label selector. It works very similiar to the ServiceMonitor resource (just without an actual Service resource). You can use the PodMonitor resource if your application does not need a Service resource (like some exporters) for any other reason. The pods should run in the same namespace as the PodMonitor is defined. Here is an example for a PodMonitor with a corresponding pod:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: my-pods
  namespace: my-app
  labels:
    prometheus.nine.ch/myprom01: scrape
spec:
  selector:
    matchLabels:
      application: my-app
  endpoints:
    - port: web
apiVersion: v1
kind: Pod
metadata:
  labels:
    application: my-app
  name: my-app
  namespace: my-app
spec:
  containers:
    - image: mycompany/example-app
      name: app
      ports:
        name: web
        containerPort: 8080

Based on the given PodMonitor resource the prometheus-operator will generate a scrape config which scrapes the shown pod "my-app" on port 8080 for metrics.

Prometheus will create a job for every ServiceMonitor or PodMonitor resource you define. It will also add a job label to all scraped metrics which have been gathered in the corresponding job. This can be used to find out from which services or pods a given metric has been scraped.

Use ScrapeConfig to scrape an external target​

The ScrapeConfig CRD can be employed to scrape targets outside the Kubernetes cluster or to create scrape configurations that are not achievable with higher-level resources such as ServiceMonitor or PodMonitor. Currently, ScrapeConfig supports a limited set of service discovery mechanisms.

Although numerous options are available (for a comprehensive list, refer to the API documentation), we currently only support static_config and http_sd configurations. The CRD is continually evolving (for now at the v1alpha1 stage), with new features and support for additional service discoveries being added regularly. We need to carefully determine which fields will be useful and need to be maintained in the long term.

note

Adding targets to your Prometheus instance can impact resource usage. As the number of targets or the cardinality of metrics increases, it may become necessary to scale up the resources or number of management nodes in your cluster to accommodate the increased resource demands.

static_config example​

The following example provide basic configuration and do not cover all supported options. For example, to scrape the target located at http://prometheus.demo.do.prometheus.io:9090, use the following configuration:

apiVersion: monitoring.coreos.com/v1alpha1
kind: ScrapeConfig
metadata:
  name: my-static-config
  namespace: my-namespace
  labels:
    prometheus.nine.ch/myprom01: scrape
spec:
  staticConfigs:
    - labels:
        job: prometheus
      targets:
        - prometheus.demo.do.prometheus.io:9090
note

The target must be specified as a hostname, not as an HTTP(S) URL. For instance, to scrape the target located at http://prometheus.demo.do.prometheus.io:9090, you should enter prometheus.demo.do.prometheus.io:9090 in the targets field.

For further details, refer to the Configuration and the API documentation.

http_sd example​

HTTP-based service discovery provides a more generic way to configure static targets and serves as an interface to plug in custom service discovery mechanisms.

It fetches targets from an HTTP endpoint containing a list of zero or more static_configs. The target must reply with an HTTP 200 response. The HTTP header Content-Type must be application/json, and the body must be valid JSON. The answer must be UTF-8 formatted. If no targets should be transmitted, HTTP 200 must also be emitted, with an empty list []. Target lists are unordered. See Requirements of HTTP SD endpoints for more information. In general, the content of the answer is as follows:

[
  {
    "targets": [ "<host>", ... ],
    "labels": {
      "<labelname>": "<labelvalue>", ...
    }
  },
  ...
]

Example response body:

[
  {
    "targets": ["prometheus.demo.do.prometheus.io:9090"],
    "labels": {
      "job": "prometheus",
      "__meta_test_label": "test_label1"
    }
  }
]
note

The URL to the HTTP SD is not considered secret. The authentication and any API keys should be passed with the appropriate authentication mechanisms. Prometheus supports TLS authentication, basic authentication, OAuth2, and authorization headers.

The endpoint is queried periodically at the specified refresh interval.

The whole list of targets must be returned on every scrape. There is no support for incremental updates. A Prometheus instance does not send its hostname and it is not possible for a SD endpoint to know if the SD requests is the first one after a restart or not.

Each target has a meta label __meta_url during the relabeling phase. Its value is set to the URL from which the target was extracted.

A simple exmple:

apiVersion: monitoring.coreos.com/v1alpha1
kind: ScrapeConfig
metadata:
  name: my-http-sd
  namespace: my-namespace
  labels:
    prometheus.nine.ch/myprom01: scrape
spec:
  httpSDConfigs:
    - url: http://my-external-api/discovery
      refreshInterval: 15s

Prometheus caches target lists and continues to use the current list if an error occurs while fetching an updated one. However, the targets list is not preserved across restarts. Therefore, it is crucial to monitor your HTTP service discovery (HTTP SD) endpoints for downtime. During a Prometheus restart, which may occur during our regular maintenance window, the cache will be cleared. If the HTTP SD endpoints are also down at this time, you may lose the endpoint target list. For more information, refer to the Requirements of HTTP SD endpoints documentation.

For further details, refer to the Configuration and the API documentation.

Querying for metrics​

You can use PromQL to query for metrics. There are some examples on the official Prometheus page. Querying can be done by using Grafana in the explore view. When using Grafana please make sure to select the data source matching your Prometheus instance. The data source name will be <YOUR PROMETHEUS NAME>/<YOUR ORG NAME>/prometheus.

Adding rules to Prometheus​

Prometheus supports two kinds of rules: recording rules and alerting rules. Both have a similar syntax, but a different use case.

Recording rules can be used to calculate new metrics from already existing ones. This can be useful if you use computationally expensive queries in dashboards. To speed them up you can create a recording rule which will evaluate the query in a defined interval and stores the result as a new metric. You can then use this new metric in your dashboard queries.

Alerting rules allow you to define alert conditions (based on PromQL). When those conditions are true, Prometheus will send out an alert to the connected Alertmanager instances. Alertmanager will then send notifications to users about alerts.

When creating alerting or recording rules, please make sure to add the prometheus.nine.ch/<your prometheus name>: scrape label with the name of your Prometheus instance. This will assign the created rule to your Prometheus instance.

The following example alerting rule will alert once a job can not reach the configured pods (targets) anymore:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  labels:
    prometheus.nine.ch/myprom01: scrape
    role: alert-rules
  name: jobs-check
spec:
  groups:
    - name: ./example.rules
      rules:
        - alert: InstanceDown
          expr: up == 0
          for: 5m
          labels:
            severity: Critical
          annotations:
            summary: "Instance {{ $labels.instance }} down"
            description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes."

This alerting rule definition will trigger an alert once an up metric gets a value of 0. The up metric is a special metric as it will be added by Prometheus itself for every job target (pod). Once a pod can not be scraped anymore, the corresponding up metric will turn to 0. If the up metric is 0 for more than 5 minutes (in this case), Prometheus will trigger an alert. The specified "labels" and "annotations" can be used in Alertmanager to customize your notification messages and routing decisions.

The full spec for the PrometheusRule definition can be found here.

Here is an example of a recording rule:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  labels:
    prometheus.nine.ch/myprom01: scrape
    role: recording-rules
  name: cpu-per-namespace-recording
spec:
  groups:
    - name: ./example.rules
      rules:
        - record: namespace:container_cpu_usage_seconds_total:sum_rate
          expr: sum(rate(container_cpu_usage_seconds_total{job="kubelet", metrics_path="/metrics/cadvisor", image!="", container!="POD"}[5m])) by (namespace)

This recording rule will create a new metric called namespace:container_cpu_usage_seconds_total:sum_rate which shows the sum of used CPU of all containers per namespace. This metric can easily be shown in a Grafana dashboard to have an overview about the CPU usage of all pods per namespace.

The kubernetes-mixins project contains sample alerts and rules for various exporters. It is a good place to get some inspiration for alerting and recording rules.

Video Guide​

Checkout our video guide series for GKE Application Monitoring. While the videos are done on our GKE product, the concepts are the same.