Skip to main content

What is zstd?

zstd is a modern compression algorithm that hast been developed at Facebook which Nine is using for the compression of database dumps and log files.

Compared to gzip, that uses the deflate algorithm, zstd usually offers a better compression, depending on the data that should be compressed, but at the same time offers a way faster compression. In some cases the compression might be a bit worse, but still way faster.

Besides being much faster zstd also uses way less CPU cycles. Where gzip usually uses one core to 100%, zstd ranges between 20 and 30% usage and therefore has a way smaller footprint.

An efficient algorithm is useful on virtualized infrastructures, especially during night times when there are a lot of maintenance tasks going on. The infrastructure will be bothered less by backup tasks such as compressing log files and database dumps and therefore the number of concurrent processes will decrease, ultimately leading to more resources for your nightly running cron jobs or web accesses.

Of course you as a customer can profit from the modern algorithm as well. We advise to compress your own archives, backups and dumps with zstd as well. The needed tools are available on our systems:

Compression: zstd myfile

  • Option: --rm -> removes the source after compression. This isn´t active as default and should be activated, otherwise the source file won´t get deleted

Decompression: unzstd myfile.zst

  • Option: --rm -> removes the source after compression. This isn´t active as default and should be activated, otherwise the source file won´t get deleted. Use --rm if you don´t need the archive anymore after extracting it.

Compression level:

  • Option: -1 to -19 -> defines how strong the compression will be. Standard is "-3". -1 offers weak compression and very high speed, while -19 will offer very strong compression at the cost of low speed. The best compromise is usually in the range between -3 and -9, depending on the amount and type of data.

zstd supports the usage of a dictionary, which can further enhance the compression factor, especially when compressing small amounts of data. That dictionary has to be present when compressing as well as when the archive should be decompressed, which can get a burden because you have to carry around the dictionary. If you don´t have access to dictionary or have lost it, you won´t be able to to decompress your archives anymore. We therefore advice to not use / build your own dictionary.

To decompress zstd archives on windows clients you can use a modified version of 7zip, available here: https://mcmilk.de/projects/7-Zip-zstd/

More information regarding zstd can be found here: https://facebook.github.io/zstd/