Skip to main content

Loki

Loki allows you to view and query logs of your containers using Grafana Loki.

Details​

Loki is a log aggregation system inspired by Prometheus. It does not index the contents of the logs, but rather a set of labels for each log stream. The logs are persisted by default for 30 days, but you can change that to whatever you want.

Availability​

Loki is available as an optional service for NKE and it can be deployed using Cockpit. In order to ship logs from an NKE cluster to Loki you will also need to create a Promtail instance on the Kubernetes Clusters page pointing to your Loki instance.

Usage​

Loki can be accessed by using the Grafana Web UI. You can find the usage instructions of Grafana here.

Labelling your pods​

If your pod is part of a deployment, statefulset or another controller, it will automatically be picked up by Loki, no matter what labels are set. We recommend using these common labels to easily find your logs.

If you run a single pod, you will need to set one of these labels to ensure Loki will pick up your logs.

  • app
  • name

Querying Logs with LogQL​

The query language used in Loki is called LogQL. To start querying your logs, head to the Grafana UI and click on Explore in the sidebar.

A LogQL query consists of two parts: log stream selector, and a search expression. A stream is selected by supplying one or more labels, for example:

{app="nginx", name=~"frontend.+"}

To search for a certain string in the results, you can use a search expression. This can be just text matching by using |= or a regex expression by using |~. And by using a ! instead of the pipe, the expression can be negated. Here are some examples:

{app="nginx"} |= "GET"
{app="nginx"} |~ "200|201|202"
{app="nginx"} != "GET"
{app="nginx"} !~ "200|201|202"

For more details, please refer to the Loki documentation.

Pushing custom Logs​

If you have pods which store logs in files rather than writing them to STDOUT, you can use any Loki client to push logs to it. Below, there's an example what this could look like. In the example we are using fluent-bit with the Loki plugin as a sidecar to a Nginx container to send logs to Loki. Please make sure to replace <LOKI_SERVICE_ADDRESS> with your specific address. The log path, format and labels are passed to fluent-bit as environment variables defined in the pod spec. More information about Fluent Bit Loki plugin.

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-loki
data:
  fluent-bit.conf: |-
    [INPUT]
        Name tail
        Path ${LOG_PATH}
    [Output]
        Name loki
        Match *
        Url http://<LOKI_SERVICE_ADDRESS>:3100/loki/api/v1/push
        BatchWait 1
        BatchSize 1001024
        Labels {app="${APP_LABEL}",pod="${POD_NAME}",namespace="${POD_NAMESPACE}"}
        LineFormat ${LOG_FORMAT}
        LogLevel info
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      volumes:
        - name: fluent-bit-config
          configMap:
            name: fluent-bit-loki
        - name: logs
          emptyDir: {}
      containers:
        - name: nginx
          image: nginx:1.7.9
          ports:
            - containerPort: 80
          volumeMounts:
            - name: logs
              mountPath: /var/log/nginx
        - name: fluent-bit-loki
          image: grafana/fluent-bit-plugin-loki:2.5.0-amd64
          volumeMounts:
            - name: fluent-bit-config
              mountPath: /fluent-bit/etc
            - name: logs
              mountPath: /var/log/nginx
          env:
            - name: APP_LABEL
              value: nginx
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: LOG_PATH
              value: /var/log/nginx/*.log
            - name: LOG_FORMAT
              value: key_value

Pushing external logs​

If you want to push logs from external systems (like an external Kubernetes cluster) to your Loki instance, please open a new ticket at the Nine helpdesk, and we will provide you with the required data.