Tern

compress

The compress module compresses bytes with zstd (fast to compress, very fast to decompress, and smaller than gzip at the same speed) and with gzip, zlib and raw deflate, for the files and protocols that use them (.gz files, HTTP Content-Encoding, PNG, zip entries). Both are built into Tern (zstd's official library and libdeflate, vendored), so there is nothing to install, and they're linked into a program only when it imports compress. They work on every target, WebAssembly included.

import compress

fn main():
    text = "a line of a log file, much like the one before it\n" * 1000
    packed = compress.zstd(text, 3)
    print(len(text), "->", len(packed))
    match compress.unzstd(packed):
        Ok(back):
            print(back == text)
        Err(e):
            print(e)
Output
50000 -> 70
true

Functions

FunctionWhat it does
compress.zstd(data str, level int) strOne zstd frame holding data, with its size in the frame header.
compress.unzstd(data str) Result[str, str]The bytes of every zstd frame in data, in order.
compress.unzstd_max(data str, max_out int) Result[str, str]The same, but Err as soon as the output passes max_out bytes.
compress.zstd_bound(n int) intThe most bytes compress.zstd of n bytes can produce.
compress.gzip(data str, level int) strOne gzip member (what a .gz file holds).
compress.gunzip(data str) Result[str, str]The bytes of every gzip member in data, in order.
compress.gunzip_max(data str, max_out int) Result[str, str]The same, with a size limit.
compress.zlib(data, level), unzlib(data), unzlib_max(data, max_out)A zlib stream (HTTP's deflate encoding, PNG).
compress.deflate(data, level), inflate(data), inflate_max(data, max_out)Raw deflate: no header or checksum (zip entries).
compress.error() strWhy the last decompression failed ("" after a success).

Both sides take and return a str, which holds any bytes, zeros included: a file read with fs.read, a request body, a [byte] converted with .to_str().

Levels

Level 3 is zstd's default and a good choice for most data. Levels 1 to 19 trade speed for size (19 is several times slower to compress than 3 and a little smaller); 20 to 22 use much more memory. Negative levels are faster still and compress less. A level outside zstd's range is clamped to it, and every level decompresses at the same speed.

Decompressing

unzstd reads frames whether or not they record their size: the frames the zstd command writes while streaming from a pipe, other libraries' streaming encoders, and several frames one after another (it returns their bytes joined). Input that isn't whole zstd frames gives an Err:

import compress

fn main():
    print(compress.unzstd("not zstd at all"))
    packed = compress.zstd("hello, hello, hello", 3)
    print(compress.unzstd(packed[..len(packed) - 2]))
Output
Err("compress: unzstd: not zstd data")
Err("compress: unzstd: truncated zstd data")

Untrusted input

A few bytes of zstd can expand to gigabytes. When the data comes from outside (a request body, an upload), use unzstd_max with the largest result you are willing to hold. It stops as soon as the output passes the limit, before it is all in memory:

import compress

fn main():
    bomb = compress.zstd("0" * 10_000_000, 19)
    print(len(bomb))
    print(compress.unzstd_max(bomb, 1_000_000))
Output
326
Err("compress: unzstd: too large")

Limits

A str holds up to 4GB, so that is the most either side can be. In a task, a call runs on the task's thread: a very large input holds up that core's other tasks briefly, as a big file read does. Wrap it in task.blocking if that matters.

gzip, zlib and deflate

import compress

fn main():
    page = "<li>an item of a long list</li>\n" * 500
    gz = compress.gzip(page, 6)
    print(len(page), "->", len(gz))
    print(compress.gunzip(gz) == Ok(page))
    two = compress.gzip("first ", 6) + compress.gzip("second", 1)
    print(compress.gunzip(two))
    print(compress.gunzip("plain text"))
    print(compress.gunzip_max(gz, 1000))
Output
16000 -> 107
true
Ok("first second")
Err("compress: gunzip: not gzip data")
Err("compress: gunzip: too large")

Levels go from 1 (fastest) to 12 (smallest, slowest); 6 is what gzip uses by default and a good choice, and 0 stores the data uncompressed. Out of range is clamped. gunzip reads every member of a file made by joining .gz files (cat a.gz b.gz), as gzip -d does, and ignores zeros after the last one; a bad checksum, data cut short, or anything else after the end is an Err. What gzip, zlib and deflate write, other gzip and zlib programs read, and the reverse (the run tests include output from gzip -9 and Python's zlib).

Each thread keeps its compressor between calls (making one costs more than compressing a few KB), so compressing many small pieces, like HTTP responses, is cheap.

Speed

bench/std compresses ~40 MB of log lines and 2 KB pieces of them, against Go's compress/gzip and Rust's flate2 (its default backend, miniz_oxide, and zlib-rs):

TernGoRust flate2Rust flate2 + zlib-rs
gzip, level 1623 MB/s (7.12 MB)327 (7.89 MB)491 (8.43 MB)536 (9.78 MB)
gzip, level 6227 MB/s (5.55 MB)89 (5.65 MB)128 (5.65 MB)181 (5.94 MB)
gunzip2,936 MB/s6239631,476
gzip of a 2 KB piece14.3 us30.715.918.3
peak memory89 MB201 MB208 MB214 MB

Tern's output is also the smallest at both levels.

Source: docs/std/compress.md. Every example on this page is compiled and run by tern x test docs.