Skip to main content

Ingress

The ingress system of Kubernetes is specifically designed to route external HTTP and HTTPS traffic into the cluster. It is composed of the ingress resource itself and an ingress controller which implements the needed logic. We offer a managed controller in the form of the HAProxy ingress controller that you can choose to deploy to your NKE cluster in Cockpit. You can control various features by adding annotations to your ingress object.

Availability​

The HAProxy ingress controller is available as an optional service for NKE. It can be deployed on an existing NKE cluster using Cockpit.

warning

The nginx ingress controller is deprecated and we will be completely shutting down support and monitoring for all Nginx ingresses on October 1st.

Please migrate your Ingress resources to use the HAProxy ingress controller before that date. Follow the step-by-step migration guide.

Usage​

The basic usage and structure of an ingress resource is documented in the official Kubernetes documentation.

Ingress DNS Name​

We provide a DNS name which will always point to your HAProxy ingress controller's IP. You can use it to point your own domain hostnames to the ingress. DNS is already set up, you will find the name in the HAProxy ingress view in Cockpit.

To use it just create an Alias or CNAME record in your own domain and point it to our provided ingress DNS.

Wildcard DNS Name​

Additionally we also create a wildcard DNS record *.<ingress-dns-name>. It is meant for quick application tests in development. You can use any hostname of that wildcard zone in your ingress resources to quickly get something up and running without requiring to mess with DNS records.

IngressClass​

To make use of our ingress controller, set the ingressClassName field in your Ingress resource to the ingress class of your HAProxy controller, which defaults to haproxy. You can change the ingress class name when you deploy the controller in Cockpit, and you can also configure whether it should be the cluster's default ingress class. If you marked the controller as the default class, you can omit the ingressClassName field.

It is possible to deploy multiple HAProxy ingress controllers per NKE cluster, each with its own ingress class. Use the ingressClassName field to select which controller handles a given ingress resource.

The deprecated nginx ingress controller is still available and uses the ingress class that was configured when it was deployed. It will be removed in a future release.

Automatic TLS Certificates​

NKE ships with cert-manager and Let's Encrypt preconfigured, so securing an ingress with free TLS certificates is as simple as adding an annotation.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    # this tells cert-manager to issue a certificate for
    # all hosts specified in the tls section
    kubernetes.io/tls-acme: "true"
  name: hello-ingress
spec:
  ingressClassName: haproxy
  rules:
    - host: app.example.org
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: hello-ingress
                port:
                  number: 80
  tls:
    - hosts:
        - app.example.org
      secretName: hello-ingress

Access Logs​

The access logs of your ingress requests can be viewed in your Grafana instance in the Loki Explore view. The ingress logs are available under the label app_kubernetes_io_name="haproxy-ingress". To only get the logs of a specific ingress instance, you can filter by using the additional label ingress. The label is in the schema of <namespace>-<ingress-name>-<backend-port>. Here's an example query to get all the logs of the ingress frontend with the port 80 in the namespace shop-prod:

{app_kubernetes_io_name="haproxy-ingress", ingress="shop-prod-frontend-80"}

Additionally the ingress logs can be filtered by these labels:

  • method the HTTP method of a request
  • status the HTTP status code of the request

For more information on the usage of Loki, refer to the specific support article.

HAProxy Ingress Features​

The HAProxy ingress controller provides many features like rate limiting, IP whitelisting, temporary or permanent redirects, etc. All of the configuration keys which can be used to control those features can be found in the official HAProxy ingress controller documentation.

Documentation for the most used features can be found below.

Basic authentication​

You can add basic authentication to your ingress resource by providing the credentials in a Kubernetes secret. Here are some instructional steps:

  1. set some env variables for easier processing

    USERNAME=<YOUR USERNAME>
    SECRET_NAMESPACE=<THE NAMESPACE FOR THE SECRET>
    INGRESS_NAMESPACE=<THE NAMESPACE OF YOUR INGRESS RESOURCE>
    INGRESS=<THE NAME OF YOUR INGRESS RESOURCE>

  2. create the Kubernetes secret which contains the credentials for basic auth. It can also be created in a different namespace than your ingress resource is stored. You will need the mkpasswd tool installed locally (can be found in the whois package in Debian/Ubuntu).

    kubectl create secret generic basic-auth-secret --namespace=$SECRET_NAMESPACE --from-literal=auth=$USERNAME:$(mkpasswd -m sha-512)

  3. add some annotations to your ingress object

    kubectl --namespace=$INGRESS_NAMESPACE annotate ingress $INGRESS haproxy-ingress.github.io/auth-secret=$SECRET_NAMESPACE/basic-auth-secret
    kubectl --namespace=$INGRESS_NAMESPACE annotate ingress $INGRESS haproxy-ingress.github.io/auth-realm='Authentication required'

Rate limiting​

You have various ways of putting rate limits on your ingresses. You can limit requests per second using the haproxy-ingress.github.io/limit-rps annotation or limit concurrent connections using the haproxy-ingress.github.io/limit-connections annotation. All available options are documented in the official HAProxy ingress docs.

Temporary and persistent redirects​

To enable a redirect to another URL for your ingress you can use the following annotation:

haproxy-ingress.github.io/redirect-to: <YOUR URL>

The redirect will use the HTTP status code of 302 (temporary) by default. If you want to change the status code, for example to 301 for a permanent redirect, use:

haproxy-ingress.github.io/redirect-to-code: "301"

HTTPS redirect​

If TLS is enabled for the given ingress, the HAProxy ingress controller will automatically redirect to the equivalent HTTPS URL of the ingress. To disable this redirect use:

haproxy-ingress.github.io/ssl-redirect: "false"

IP whitelisting​

You can allowlist the IP addresses which are allowed to connect to your ingress resource. You can specify them in CIDR notation in the following annotation:

haproxy-ingress.github.io/allowlist-source-range: <YOUR CIDR RANGE>

Custom default backend​

The default backend is responsible for showing a 404 error page if a request arrives on the HAProxy ingress controller for which no ingress rule was specified. You can create your own custom default backend (+ kubernetes service) and refer to it on your ingress object.

The default backend only has 2 requirements:

  • it needs to serve a 404 page/code on the path /
  • it needs to serve a 200 HTTP code on the path /healthz

Once you built and deployed your default backend service in the same namespace as your ingress resource you can refer to it via the following annotation on your ingress:

haproxy-ingress.github.io/default-backend: <SERVICE NAME OF YOUR DEFAULT BACKEND>

Static IPs for ingress and egress​

The IP of every ingress controller is static. This means all ingress resources using that controller will share that IP. In addition, it is possible to deploy multiple ingress controllers per NKE cluster and you can choose the controller with the ingressClassName field in the ingress resource.

While ingress IPs remain static, it's important to note that, by default, egress IPs are currently dynamic due to operational considerations. Specifically, during node replacements, such as underlying OS version upgrades, NKE initiates the creation of a new node. The existing workloads are seamlessly moved to the new node, which then inherits a new IP address. As a result, the egress IPs are subject to change in this process. If you need a static egress IP for your cluster workloads, you can make use of our static-egress feature.

IP ranges for egress​

See Subnets in each Location for the current egress IP ranges.

Deprecated: Nginx Ingress Controller​

warning

The nginx ingress controller is deprecated and will be removed in a future release. Please migrate your Ingress resources to the HAProxy ingress controller documented above.

Show deprecated nginx ingress documentation

IngressClass​

To use the deprecated nginx ingress controller, set the ingressClassName field in your Ingress resource to its configured ingress class, that is, the class that was set when the controller was deployed.

Access Logs​

The nginx ingress logs are available under the Loki label app="ingress-nginx". Example query:

{app="ingress-nginx",ingress="shop-prod-frontend-80"}

Nginx Ingress Features​

All available annotations can be found in the official nginx ingress controller documentation.

Basic authentication​

kubectl --namespace=$INGRESS_NAMESPACE annotate ingress $INGRESS nginx.ingress.kubernetes.io/auth-type=basic
kubectl --namespace=$INGRESS_NAMESPACE annotate ingress $INGRESS nginx.ingress.kubernetes.io/auth-secret=$SECRET_NAMESPACE/basic-auth-secret
kubectl --namespace=$INGRESS_NAMESPACE annotate ingress $INGRESS nginx.ingress.kubernetes.io/auth-realm='Authentication required'

Rate limiting​

All available options are documented in the official nginx ingress docs.

Temporary and persistent redirects​

# Temporary redirect (HTTP 302)
nginx.ingress.kubernetes.io/temporal-redirect: <YOUR URL>
# Permanent redirect (HTTP 301)
nginx.ingress.kubernetes.io/permanent-redirect: <YOUR URL>

HTTPS redirect​

nginx.ingress.kubernetes.io/ssl-redirect: "false"

IP whitelisting​

nginx.ingress.kubernetes.io/whitelist-source-range: <YOUR CIDR RANGE>

Caching​

The nginx ingress controllers allows to enable basic caching of backend resources, which can be particularly useful for static content. You can enable caching on specific ingress resources by setting the following annotations in your ingress definition (metadata.annotations):

nginx.ingress.kubernetes.io/proxy-buffering: "on"
nginx.ingress.kubernetes.io/configuration-snippet: |
  proxy_cache static-cache;
  proxy_cache_valid 10m;
  proxy_cache_use_stale error timeout updating http_404 http_500 http_502 http_503 http_504;
  proxy_cache_bypass $http_x_purge;
  add_header X-Cache-Status $upstream_cache_status;

In this example, the cache is invalidated after 10 minutes, and only HTTP status codes 200, 301 and 302 are cached. If another behaviour should be desired, proxy_cache_valid can also take a list of status codes in front of the time:

proxy_cache_valid 404 1m;

would cache 404 responses for one minute. There is also the special code any, which can be specified to cache any responses. Additionally, multiple proxy_cache_valid statements can be added on one ingress to specify different cache times for different status codes.

The default cache size is 100MB. Should you have other requirements, please do not hesitate to get in contact with us.

To check that the caching worked, you can use cURL to inspect the header of the returned resource. For that, you will need to execute the command twice, and at the second request, the resource should be cached:

curl --head <URL>

should print the header x-cache-status: HIT.

If you want to enable caching only for a sub-path of your application (for example an endpoint /static), you will need to create two separate ingress resources. To further configure caching, you can use the proxy_cache options that are valid for location blocks. See the nginx proxy_cache module documentation for available options.

Custom default backend​

The default backend is responsible for showing a 404 error page if a request arrives on the nginx ingress controller for which no ingress rule was specified. You can create an own custom default backend (+ kubernetes service) and refer to it on your ingress object.

The default backend only has 2 requirements:

  • it needs to serve a 404 page/code on the path /
  • it needs to serve a 200 HTTP code on the path /healthz

The implementation of the default nginx ingress controller backend can be found in the ingress-gce 404-server source.

Once you built and deployed your default backend service in the same namespace as your ingress resource you can refer to it via the following annotation on your ingress:

nginx.ingress.kubernetes.io/default-backend: <SERVICE NAME OF YOUR DEFAULT BACKEND>
Additional custom error pages​

To be able to additionally display custom error pages on the default backend (for example if the service your ingress points to is not available) you can use the following annotation:

nginx.ingress.kubernetes.io/custom-http-errors: <ERROR CODES> # for example: "404,415,503"

The nginx ingress controller will forward error information via certain HTTP headers to your default backend, which then can return the best possible error representation. More information about this can be found in the official documentation. An example of a default backend which can display custom error pages can be found in the ingress-nginx custom-error-pages repository.

Migrating From Nginx to HAProxy​

You can run the nginx and HAProxy ingress controllers side by side. This lets you migrate one ingress at a time and verify each application on HAProxy before sending production traffic to it, so no maintenance window is required.

The migration is a per-ingress process: for each nginx Ingress resource you add an equivalent HAProxy Ingress, verify it, switch DNS, and then remove the old one.

Migrating Annotations From Nginx​

If your ingress resources use nginx-specific annotations, you need to update them to their HAProxy equivalents. Here are the most common ones:

Nginx annotationHAProxy annotation
nginx.ingress.kubernetes.io/ssl-redirecthaproxy-ingress.github.io/ssl-redirect
nginx.ingress.kubernetes.io/auth-secrethaproxy-ingress.github.io/auth-secret
nginx.ingress.kubernetes.io/auth-realmhaproxy-ingress.github.io/auth-realm
nginx.ingress.kubernetes.io/whitelist-source-rangehaproxy-ingress.github.io/allowlist-source-range
nginx.ingress.kubernetes.io/temporal-redirecthaproxy-ingress.github.io/redirect-to (defaults to status code 302)
nginx.ingress.kubernetes.io/permanent-redirecthaproxy-ingress.github.io/redirect-to + haproxy-ingress.github.io/redirect-to-code: "301"
nginx.ingress.kubernetes.io/limit-rpshaproxy-ingress.github.io/limit-rps

  1. Deploy the HAProxy ingress controller for your cluster in Cockpit. Note its ingress class name (defaults to haproxy) and its ingress DNS name, both shown in the HAProxy ingress view. See Availability.

  2. Add a second Ingress resource for your application that uses the HAProxy ingress class and the equivalent HAProxy annotations, and leave the existing nginx ingress in place for now. See Migrating Annotations From Nginx for the annotation mapping.

    To find the ingress resources you still need to migrate, list them across all namespaces. The CLASS column shows which controller each one uses:

    kubectl get ingress --all-namespaces

    For example, this nginx ingress:

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: hello
      annotations:
        nginx.ingress.kubernetes.io/ssl-redirect: "true"
    spec:
      ingressClassName: nginx
      rules:
        - host: app.example.org
          http:
            paths:
              - path: /
                pathType: Prefix
                backend:
                  service:
                    name: hello
                    port:
                      number: 80

    gets an HAProxy counterpart with the translated ingressClassName and annotations. While testing, point it at a hostname under the HAProxy wildcard DNS name so you don't have to change your production DNS yet:

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: hello-haproxy
      annotations:
        haproxy-ingress.github.io/ssl-redirect: "true"
    spec:
      ingressClassName: haproxy
      rules:
        # a test hostname under the HAProxy wildcard DNS name
        - host: hello.<haproxy-ingress-dns-name>
          http:
            paths:
              - path: /
                pathType: Prefix
                backend:
                  service:
                    name: hello
                    port:
                      number: 80

  3. Verify that your application is reachable through the HAProxy ingress using the test hostname:

    curl http://hello.<haproxy-ingress-dns-name>/

  4. Switch your DNS records to the HAProxy ingress. Update the CNAME or Alias record for your domain so it points to the HAProxy ingress DNS name instead of the nginx one. You can then replace the test hostname in the HAProxy ingress with your production hostname. See Ingress DNS Name.

  5. Remove the old nginx Ingress resource once all traffic is served through HAProxy:

    kubectl --namespace=<namespace> delete ingress <nginx-ingress-name>

  6. Remove the old nginx ingress controller in Cockpit once no Ingress resources reference its ingress class anymore. If the nginx controller was your cluster's default ingress class, set the HAProxy controller as the new default in Cockpit so ingresses without an explicit ingressClassName keep working.