Skip to main content

Backup and Restore

Backup & Restore is a service of NKE allowing for regular backups and recovery of cluster data and configuration.

Details​

Customers of NKE need peace of mind that their cluster configuration and Persistent Volume Claim (PVC) data is backed up and can be made available when needed, for security and disaster recovery. Therefore Nine regularly creates automated backups and on customer request engages in recovery and deployment of those backups.

Data of Persistent Volumes is snapshotted and stored redundantly across our virtualization infrastructure. Kubernetes resource backups are saved in an Object Storage Bucket.

Availability​

Backup/Restore is available as standard with NKE.

Usage​

  • Backups of data and configuration will automatically be taken nightly
  • Backups are retained for 30 days by default
  • You can restore namespaces yourself

Restoring namespaces​

To create restores, the velero utility needs to be installed locally.

Please make sure that your current kubecontext contains your NKE cluster by following the documented cluster login steps.

You can restore a complete namespace yourself. First you need to find the backup from which you want to restore. Execute the following command to list all backups:

velero backup get -n nine-system

Once you found the backup from which you want to restore, you can restore the whole content of one namespace into another one by using:

velero restore create -n nine-system --from-backup <identifiedBackup> --include-namespaces <namespaceToBeRestored> --namespace-mappings <namespaceToBeRestored>:<targetNamespace>

If you want to restore the content of an existing namespace, you either delete the target namespace before restoring it or you delete all existing resources which would be restored by velero. Velero does not overwrite any existing resources.

velero restore create -n nine-system --from-backup <identifiedBackup> --include-namespaces <namespaceToBeRestored>

Backups of Read Write Many (RWX files) volumes​

caution

The backups of RWX Persisten Volume Claims (PVC) are coupled to the PVC itself. If you accidentially delete the PVC, the corresponding backups will also be removed. To prevent an accidential deletion you can make use of our deletion protection feature.

For RWX volumes, the restore process is slightly different. These volumes use an integrated snapshotting mechanism that runs outside of the usual Velero schedule.

The schedule is fixed and defined as follows:

  • Hourly: a new snapshot is created every hour and kept for 24h.
  • Daily: a new snapshot is created daily and kept for 7 days.
  • Weekly: a new snapshot is created every week on Sunday and kept for 4 weeks.
  • Monthly: a new snapshot is created every first day of the month and kept for 3 months.

The snapshot data can be accessed from within the filesystem of the volume in in a directory named .snapshot in the root of the volume. To restore file(s) from older snapshots, you can either exec into an existing pod that mounts the volume or create a temporary pod to do the restore.

# create a temporary pod, change the <pvc-name> to the PVC you want to access.
$ kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: files-snapshots
spec:
  containers:
  - args:
    - /bin/sh
    image: instrumentisto/rsync-ssh
    name: rsync
    stdin: true
    stdinOnce: true
    tty: true
    volumeMounts:
    - mountPath: /data
      name: files-pvc
  volumes:
  - name: files-pvc
    persistentVolumeClaim:
      claimName: <pvc-name>
EOF
$ kubectl exec -ti files-snapshot sh
# all the PVC data is mounted at /data, so you can now list all your snapshots.
$ ls -l /data/.snapshot
total 14
drwxrwxrwt    2 root     root             3 Nov 14 15:00 afs-auto-snap_hourly-2023-11-15-1100
drwxrwxrwt    2 root     root             3 Nov 14 15:00 afs-auto-snap_hourly-2023-11-15-1200
drwxrwxrwt    2 root     root             3 Nov 14 15:00 afs-auto-snap_hourly-2023-11-15-1300

Once your temporary pod is up and running you can simply copy files from an older snapshot to /data or use the rsync command to sync larger directories.