Object Storage Client Tools

Besides integrations into your development framework, there are dozens of tools to access buckets. We present configurations for the most common tools here. For more details, it is best to consult the documentation for the respective software directly.

Backend v1 vs. v2

Depending on which backend version you are using, there are some slight differences in configuration.

API Region

For bucket v1, the region matches the physical location of the service, so if your bucket is in nine-cz42, the region parameter will also need to be nine-cz42. This changes with bucket v2, where the region always needs to be us-east-1, regardless of the physical location. The reason for this is that us-east-1 is the default region in AWS and many client tools set this out of the box. This improves compatibility as some clients don't even allow customizing the region beyond what AWS is offering.

Physical Location	v1 Region	v2 Region
nine-cz42	nine-cz42	us-east-1
nine-es34	nine-es34	us-east-1

API Hostname

The hostname for accessing the S3 API differs depending on the backend version.

Physical Location	v1 Hostname	v2 Hostname
nine-cz42	cz42.objectstorage.nineapis.ch	cz42.objects.nineapis.ch
nine-es34	es34.objectstorage.nineapis.ch	es34.objects.nineapis.ch

Host and path style Bucket access

For v2 buckets it's now also possible to access the bucket via the hostname directly instead of just the path-style of v1 buckets:

Path-style: https://cz42.objects.nineapis.ch/bucket-name

Host-style: https://bucket-name.cz42.objects.nineapis.ch

s3cmd

s3cmd is a popular command line tool for managing data in object storage buckets. The tool is designed for AWS S3, but works fine with other compatible systems, such as our Object Storage. To configure it, you need details about your bucket and the corresponding user. Make sure that the user has access to the bucket.

This is an example configuration, which should be in a file ~/.s3cfg. Adjust the values according to the information for your user and bucket.

[default]
access_key = 6aaf50b17357446bb1a25a6c93361569
secret_key = fcf1c9c6bc5c4384a4e0dbff99d3cc52
host_base = cz42.objects.nineapis.ch
host_bucket = cz42.objects.nineapis.ch
use_https = True

After that you can start uploading files.

s3cmd put image01.jpg s3://my-bucket

rclone

rclone is commonly used for backing up data to an S3 compatible storage. However, there are numerous other backends for various other storages.

This is an example configuration which should be in a file ~/.config/rclone/rclone.conf. Adjust the values according to the information for your user and bucket.

[nine-cz42]
type = s3
provider = Other
access_key_id = 6aaf50b17357446bb1a25a6c93361569
secret_access_key = fcf1c9c6bc5c4384a4e0dbff99d3cc52
endpoint = https://cz42.objects.nineapis.ch

After that, files can be uploaded from the local disk to the object storage or from other configured rclone backends.

rclone copy image01.png nine-cz42:my-bucket

Heads up! Due to a bug you might have to pass the flag --s3-no-check-bucket when using a newer version of rclone (>=v1.53.3) or you will receive permission errors when uploading files.

Or you can synchronize a local directory with the Object Storage:

rclone sync backup-dir nine-cz42:mybucket

restic

restic is an exceptional backup solution that is easy to use and has great deduplication.

Heads up! Older version of restic seems to have a problem with the Region Setting. Please use v0.12.0 or newer.

Similar to the other tools, you first need to define the s3 variables. This can either be done with environments variables or with command parameters. For this example, we go for the env variables, since this is more practical:

We add the following variables to the config file: ~/my_backup/restic.conf

export AWS_ACCESS_KEY_ID="6aaf50b17357446bb1a25a6c93361569"
export AWS_SECRET_ACCESS_KEY="fcf1c9c6bc5c4384a4e0dbff99d3cc52"
export RESTIC_REPOSITORY="s3:https://cz42.objects.nineapis.ch/bucket-etj4mwciyzuv"
export RESTIC_PASSWORD="SUPER-SECURE-PASSWORD"

In contrast to the other tools, restic works with repositories. So at first, we need to create such a repository. To do this, we need to load the env variables we set before:

$ source ~/my_backup/restic.conf

Then we can initialize the repository:

$ restic init

Now we're ready to backup some stuff:

$ restic backup /home/www-data/this_should_be_backuped

Now we want to check, what backups we already have:

$ restic snapshots
repository ff06b869 opened successfully, password is correct
ID        Time                 Host             Tags        Paths
-------------------------------------------------------------------------------------------------------------------
bb689eb9  2021-07-16 11:45:27  mysuperserver              /home/www-data/this_should_be_backuped.bin
-------------------------------------------------------------------------------------------------------------------
1 snapshots

One backup run will create one snapshot. To delete old snapshots, you can do this:

$ restic forget bb689eb9 --prune

But since you most likely won't delete all the snapshots manually, you can do this with retention policies:

$ restic forget --keep-daily 30 --keep-weekly 4 --keep-monthly 12 --keep-yearly 10 --prune

Please keep in mind, that this is just a super simple example how to backup a directory. restic is quite powerful and has a lot of options. Check out their documentation for a more detailed help: https://restic.readthedocs.io/en/stable/040_backup.html

Using Object Storage in code

Using object storage in code is a little more complicated than using file storage, luckily there are a number of libraries that you can use to make it easier. Here we will look at an example in Javascript using the minio library.

Firstly install the dependencies that we need with npm. For this example we will be getting the information for the Client from the environment via dotenv, but you could substitute this with any way which is most suitable for your use case npm install minio dotenv.

As the code examples below will be presented in Typescript format, making it easier to see what types of objects you should expect to work with, you will need to also npm install --save-dev @types/minio to use them directly.

Firstly you will need to construct a client with the correct information from the environment:

import * as Minio from "minio"

//This allows us to get environmental variables from process.env
require("dotenv").config()

// Construct the client with the correct data, passing default values if the env vars do not exist.
// Note that you must supply a key and secret to use object storage!
const client = new Minio.Client({
  endPoint: process.env.BUCKET_ENDPOINT || "cz42.objects.nineapis.ch",
  region: process.env.BUCKET_REGION || "us-east-1",
  useSSL: true,
  accessKey: process.env.BUCKET_KEY || "",
  secretKey: process.env.BUCKET_SECRET || "",
})

The corresponding .env file would look something like this:

BUCKET_ENDPOINT=cz42.objects.nineapis.ch  # or es34.objects.nineapis.ch
BUCKET_REGION=us-east-1
BUCKET_KEY=my-bucket-key-from-cockpit-or-api
BUCKET_SECRET=my-bucket-secret-from-cockpit-or-api

Now that we have a working connection to our object storage we will need some functions to read or write from it. The minio library has two ways that we can read data, we can either write it as a file to the local disk, or we can stream the contents of the file into memory. Since there is a good chance that if you are using object storage you may not have filesystem to write to, we will cover the streaming approach here. Streaming the data is more complex than writing a file to disk since you will need to deal with the stream state and your file will be passed in chunks that you need to process. The following functions would achieve this:

// Asyncronously read some data from your object storage, this returns a promise that
// either returns a Buffer of data or null if the key does not exist. Will throw on errors
async function read(bucket: string, key: string): Promise<Buffer | null> {
  return new Promise(async (resolve, reject) => {
    // Wrap everything in a try/catch because we might get an error
    // thrown before we get a stream returned
    try {
      // the the stream that we will read from. Note that you could pass
      // the stream handling as a function to getObject, but if we want to
      // easily return the data it's easier to await the stream and handle it
      const stream = await client.getObject(bucket, key)
      // we will use this array to get the chunks returned from the stream
      const chunks: any[] = []

      stream.on("data", (chunk) => {
        // when the stream reports that it has data get that data and add it to the array
        chunks.push(chunk)
      })

      stream.on("end", () => {
        // On end we have the entirety of the file in the chunks array,
        // so we make it into a buffer and return it for processing
        // If your stream was a JSON file this processing could look like:
        // JSON.parse(returnedBuffer.toString())
        const fileContent = Buffer.concat(chunks)
        resolve(fileContent)
      })

      stream.on("error", (err) => {
        // If we have an error from the stream we reject the promise instead
        // of resolving it and pass on the error we received
        reject(err)
      })
    } catch (error: any) {
      // If we got an error getting the stream for the file then handle it here
      if (error.code === "NoSuchKey") {
        // In this case we are resolving but not returning any data
        // because not having a key is not an error per-se, and it
        // is a regular use case to check if a key exists before performing
        // a write, which this would allow, as you could simply check for null
        // on resolution of the promise
        resolve(null)
      } else {
        // Every error other than NoSuchKey is treated as fatal and causes this
        // promise to be rejected
        reject(error)
      }
    }
  })
}

// Asynchronously write some data to the buffer, returns a promise true if writing works
// otherwise throws an error
async function write(bucket: string, key: string, data: any): Promise<boolean> {
  return new Promise(async (resolve, reject) => {
    // put the data to the object storage, data must be a readable stream
    // See minio docs for more info about this:
    // https://min.io/docs/minio/linux/developers/javascript/API.html#putobject-bucketname-objectname-stream-size-metadata-callback
    // But the simplest version would be JSON.stringify(my-data-string-var)
    client.putObject(bucket, key, data, (err, data) => {
      if (err) reject(err)
      else resolve(true)
    })
  })
}