Ingress

The ingress system of Kubernetes is specifically designed to route external HTTP and HTTPS traffic into the cluster. It is composed of the ingress resource itself and an ingress controller which implements the needed logic. We offer a managed controller in the form of the HAProxy ingress controller that you can choose to deploy to your NKE cluster in Cockpit. You can control various features by adding annotations to your ingress object.

Availability

The HAProxy ingress controller is available as an optional service for NKE. It can be deployed on an existing NKE cluster using Cockpit. On a vCluster, a predefined Ingress controller from the underlying NKE host cluster is available out of the box.

note

The nginx ingress controller is deprecated. For its documentation and how to migrate to HAProxy, see Nginx Ingress.

Usage

The basic usage and structure of an ingress resource is documented in the official Kubernetes documentation.

Ingress DNS Name

We provide a DNS name which will always point to your HAProxy ingress controller's IP. You can use it to point your own domain hostnames to the ingress. DNS is already set up, you will find the name in the HAProxy ingress view in Cockpit.

To use it just create an Alias or CNAME record in your own domain and point it to our provided ingress DNS.

Wildcard DNS Name

Additionally we also create a wildcard DNS record *.<ingress-dns-name>. It is meant for quick application tests in development. You can use any hostname of that wildcard zone in your ingress resources to quickly get something up and running without requiring to mess with DNS records.

IngressClass

To make use of our ingress controller, set the ingressClassName field in your Ingress resource to the ingress class of your HAProxy controller, which defaults to haproxy. You can change the ingress class name when you deploy the controller in Cockpit, and you can also configure whether it should be the cluster's default ingress class. If you marked the controller as the default class, you can omit the ingressClassName field.

It is possible to deploy multiple HAProxy ingress controllers per NKE cluster, each with its own ingress class. Use the ingressClassName field to select which controller handles a given ingress resource.

The deprecated nginx ingress controller is still available and uses the ingress class that was configured when it was deployed. It will be removed in a future release.

Automatic TLS Certificates

NKE ships with cert-manager and Let's Encrypt preconfigured, so securing an ingress with free TLS certificates is as simple as adding an annotation.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    # this tells cert-manager to issue a certificate for
    # all hosts specified in the tls section
    kubernetes.io/tls-acme: "true"
  name: hello-ingress
spec:
  ingressClassName: haproxy
  rules:
    - host: app.example.org
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: hello-ingress
                port:
                  number: 80
  tls:
    - hosts:
        - app.example.org
      secretName: hello-ingress

If your cluster runs more than one ingress controller, for example while you are migrating from nginx to HAProxy, also set the ingress class that cert-manager should use to solve the ACME HTTP-01 challenge:

metadata:
  annotations:
    kubernetes.io/tls-acme: "true"
    # solve the ACME challenge through the HAProxy ingress
    acme.cert-manager.io/http01-ingress-ingressclassname: haproxy

Without this annotation, cert-manager creates the challenge ingress on the cluster's default ingress class. If that is a different controller than the one serving your ingress, the certificate is never issued or renewed.

Access Logs

The access logs of your ingress requests can be viewed in your Grafana instance in the Loki Explore view. The access logs are written by a dedicated container of the ingress controller and are available under the labels app="haproxy-ingress" and container="access-logs":

{app="haproxy-ingress", container="access-logs"}

To narrow the logs down to a specific backend, use a line filter. Each access log line contains the target backend in the schema <namespace>_<service>_<port>. Here's an example query to get all the logs of the service frontend with the port 80 in the namespace shop-prod:

{app="haproxy-ingress", container="access-logs"} |= "shop-prod_frontend_80"

The HTTP method, status code, request path and timings are part of the log line itself, so you can filter for them with additional line filters (for example |= " 503 " to find requests that returned a 503).

For more information on the usage of Loki, refer to the specific support article.

HAProxy Ingress Features

The HAProxy ingress controller provides many features like rate limiting, IP whitelisting, temporary or permanent redirects, etc. All of the configuration keys which can be used to control those features can be found in the official HAProxy ingress controller documentation.

Documentation for the most used features can be found below.

Basic authentication

You can add basic authentication to your ingress resource by providing the credentials in a Kubernetes secret. Here are some instructional steps:

1. set some env variables for easier processing

   USERNAME=<YOUR USERNAME>
   SECRET_NAMESPACE=<THE NAMESPACE FOR THE SECRET>
   INGRESS_NAMESPACE=<THE NAMESPACE OF YOUR INGRESS RESOURCE>
   INGRESS=<THE NAME OF YOUR INGRESS RESOURCE>

2. create the Kubernetes secret which contains the credentials for basic auth. It can also be created in a different namespace than your ingress resource is stored. You will need the mkpasswd tool installed locally (can be found in the whois package in Debian/Ubuntu).

   kubectl create secret generic basic-auth-secret --namespace=$SECRET_NAMESPACE --from-literal=auth=$USERNAME:$(mkpasswd -m sha-512)

3. add some annotations to your ingress object

   kubectl --namespace=$INGRESS_NAMESPACE annotate ingress $INGRESS haproxy-ingress.github.io/auth-secret=$SECRET_NAMESPACE/basic-auth-secret
   kubectl --namespace=$INGRESS_NAMESPACE annotate ingress $INGRESS haproxy-ingress.github.io/auth-realm='Authentication required'

Rate limiting

You have various ways of putting rate limits on your ingresses. You can limit requests per second using the haproxy-ingress.github.io/limit-rps annotation or limit concurrent connections using the haproxy-ingress.github.io/limit-connections annotation. All available options are documented in the official HAProxy ingress docs.

Temporary and persistent redirects

To enable a redirect to another URL for your ingress you can use the following annotation:

haproxy-ingress.github.io/redirect-to: <YOUR URL>

The redirect will use the HTTP status code of 302 (temporary) by default. If you want to change the status code, for example to 301 for a permanent redirect, use:

haproxy-ingress.github.io/redirect-to-code: "301"

HTTPS redirect

If TLS is enabled for the given ingress, the HAProxy ingress controller will automatically redirect to the equivalent HTTPS URL of the ingress. To disable this redirect use:

haproxy-ingress.github.io/ssl-redirect: "false"

IP whitelisting

You can allowlist the IP addresses which are allowed to connect to your ingress resource. You can specify them in CIDR notation in the following annotation:

haproxy-ingress.github.io/allowlist-source-range: <YOUR CIDR RANGE>

Custom default backend

The default backend is responsible for showing a 404 error page if a request arrives on the HAProxy ingress controller for which no ingress rule was specified. You can create your own custom default backend (+ kubernetes service) and refer to it on your ingress object.

The default backend only has 2 requirements:

• it needs to serve a 404 page/code on the path /
• it needs to serve a 200 HTTP code on the path /healthz

Once you built and deployed your default backend service in the same namespace as your ingress resource you can refer to it via the following annotation on your ingress:

haproxy-ingress.github.io/default-backend: <SERVICE NAME OF YOUR DEFAULT BACKEND>

Static IPs for ingress and egress

The IP of every ingress controller is static. This means all ingress resources using that controller will share that IP. In addition, it is possible to deploy multiple ingress controllers per NKE cluster and you can choose the controller with the ingressClassName field in the ingress resource.

While ingress IPs remain static, it's important to note that, by default, egress IPs are currently dynamic due to operational considerations. Specifically, during node replacements, such as underlying OS version upgrades, NKE initiates the creation of a new node. The existing workloads are seamlessly moved to the new node, which then inherits a new IP address. As a result, the egress IPs are subject to change in this process. If you need a static egress IP for your cluster workloads, you can make use of our static-egress feature.

IP ranges for egress

See Subnets in each Location for the current egress IP ranges.

Documentation and Links

• the official kubernetes ingress documentation
• all available configuration keys of the HAProxy ingress controller