Centralized Logging with Loki

Centralized Logging allows you to view and query logs of your containers using Grafana Loki.

Details

Loki is a log aggregation system inspired by Prometheus. It does not index the contents of the logs, but rather a set of labels for each log stream. The logs are persisted for 90 days.

Availability

Centralized Logging is available as standard with nine Managed GKE.

Usage

Loki can be accessed by using the Grafana Web UI. The login details are provided on runway.

Labelling your pods

If your pod is part of a deployment, statefulset or another controller, it will automatically be picked up by Loki, no matter what labels are set. We recommend using these common labels to easily find your logs.

If you run a single pod, you will need to set one of these labels to ensure Loki will pick up your logs.

• app
• name

Querying Logs with LogQL

The query language used in Loki is called LogQL. To start querying your logs, head to the Grafana UI and click on Explore in the sidebar or use the direct link provided on runway.

A LogQL query consists of two parts: log stream selector, and a search expression. A stream is selected by supplying one or more labels, for example:

{app="nginx", name=~"frontend.+"}

To search for a certain string in the results, you can use a search expression. This can be just text matching by using |= or a regex expression by using |~. And by using a ! instead of the pipe, the expression can be negated. Here are some examples:

{app="nginx"} |= "GET"
{app="nginx"} |~ "200|201|202"
{app="nginx"} != "GET"
{app="nginx"} !~ "200|201|202"

For more details, please refer to the Loki documentation.

Pushing custom Logs

If you have pods which store logs in files rather than writing them to STDOUT, you can use any Loki client to push logs to it. Below, there's an example what this could look like. In the example we are using fluent-bit with the Loki plugin as a sidecar to an Nginx container to send logs to Loki. Please make sure to replace <LOKI_SERVICE_ADDRESS> with the address found on runway. The log path, format and labels are passed to fluent-bit as environment variables defined in the pod spec. More information about Fluent Bit Loki plugin.

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-loki
data:
  fluent-bit.conf: |-
    [INPUT]
        Name tail
        Path ${LOG_PATH}
    [Output]
        Name loki
        Match *
        Url http://<LOKI_SERVICE_ADDRESS>:3100/loki/api/v1/push
        BatchWait 1
        BatchSize 1001024
        Labels {app="${APP_LABEL}",pod="${POD_NAME}",namespace="${POD_NAMESPACE}"}
        LineFormat ${LOG_FORMAT}
        LogLevel info
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      volumes:
        - name: fluent-bit-config
          configMap:
            name: fluent-bit-loki
        - name: logs
          emptyDir: {}
      containers:
        - name: nginx
          image: nginx:1.7.9
          ports:
            - containerPort: 80
          volumeMounts:
            - name: logs
              mountPath: /var/log/nginx
        - name: fluent-bit-loki
          image: grafana/fluent-bit-plugin-loki:v0.4.0-amd64
          volumeMounts:
            - name: fluent-bit-config
              mountPath: /fluent-bit/etc
            - name: logs
              mountPath: /var/log/nginx
          env:
            - name: APP_LABEL
              value: nginx
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: LOG_PATH
              value: /var/log/nginx/*.log
            - name: LOG_FORMAT
              value: key_value

Pushing external logs

If you want to push logs from external systems (like a external kubernetes cluster) to loki, we can create a basic auth secured ingress resource which will forward traffic to your loki instance. You can then use fluent-bit or promtail to push logs. Please contact support to enable that feature.