Faster Compression by zstd

less than 1 minute read

Published:

Recently I was required to transfer a bulk of data containing millions of sparse files, where the archive and compression algorithms became a must yet bottleneck to my server. A compression algorithm with full parallelism, zstd, can better address the problem. Some tips are recorded in post.

Installation

zstd suggests install via self building.

$ wget https://github.com/facebook/zstd/releases/download/v1.5.2/zstd-1.5.2.tar.gz
$ tar -xf zstd-1.5.2.tar.gz && cd zstd-1.5.2
$ sudo make         # compile
$ sudo make install

Enable Parallelism

zstd provides elaborated commandline options. Using man zstd to have a check.

For parallelism we can simply pass --fast= option into zstd cli.

$ zstd --fast=1  # default speed
$ zstd --fast=15 # maximum speed

Use with tar Archiver

zstd is a compression implementation supports co-working with tar archiver.

$ tar -I "zstd --fast=15" -cvf dest.tar.zst srcdir # compression
$ tar -I "zstd --fast=15" -xvf src.tar.zst         # decompression

You May Also Enjoy

ARM Permission Indirection

3 minute read

Published:

In 2022, ARM introduced a new way to control memory permissions. Instead of directly encoding the permission in the Translation Table Entry (TTE), fields in the TTEs are used to index into an array of permissions specified in a register. This indirection provides greater flexibility, greater encoding density and enables the representation of new permissions.

Build Kernel Module from a Customized Kernel

1 minute read

Published:

In recent projects, we need to hack on Linux Kernel with various customization on multiple architectures. However, common distros like Fedora/Debian/Centos will not provide compatible kernel development headers (staffs under /lib/modules/$version) required by building kernel modules.

Run Debian in QEMU

3 minute read

Published:

One need we recently met was to directly trigger syscalls in QEMU. To achieve this, we have to run Linux with pre-installed dev tools such as vim and gcc, and most importantly, package managers like apt. Thanks to debootstrap, such a subsystem can be setup within minutes.

LaTeX for Fun and Profit

2 minute read

Published:

LaTeX plays an essential role in computer science academics. A decent layout leads to the entertainment of both reviewers and ourselves. Believing this, I write the following to demonstrate some tips of LaTeX best practice.