compress
The compress module compresses bytes with zstd (fast to compress, very fast to decompress, and smaller than gzip at the same speed) and with gzip, zlib and raw deflate, for the files and protocols that use them (.gz files, HTTP Content-Encoding, PNG, zip entries). Both are built into Tern (zstd's official library and libdeflate, vendored), so there is nothing to install, and they're linked into a program only when it imports compress. They work on every target, WebAssembly included.
import compress
fn main():
text = "a line of a log file, much like the one before it\n" * 1000
packed = compress.zstd(text, 3)
print(len(text), "->", len(packed))
match compress.unzstd(packed):
Ok(back):
print(back == text)
Err(e):
print(e)50000 -> 70
trueFunctions
| Function | What it does |
|---|---|
compress.zstd(data str, level int) str | One zstd frame holding data, with its size in the frame header. |
compress.unzstd(data str) Result[str, str] | The bytes of every zstd frame in data, in order. |
compress.unzstd_max(data str, max_out int) Result[str, str] | The same, but Err as soon as the output passes max_out bytes. |
compress.zstd_bound(n int) int | The most bytes compress.zstd of n bytes can produce. |
compress.gzip(data str, level int) str | One gzip member (what a .gz file holds). |
compress.gunzip(data str) Result[str, str] | The bytes of every gzip member in data, in order. |
compress.gunzip_max(data str, max_out int) Result[str, str] | The same, with a size limit. |
compress.zlib(data, level), unzlib(data), unzlib_max(data, max_out) | A zlib stream (HTTP's deflate encoding, PNG). |
compress.deflate(data, level), inflate(data), inflate_max(data, max_out) | Raw deflate: no header or checksum (zip entries). |
compress.error() str | Why the last decompression failed ("" after a success). |
Both sides take and return a str, which holds any bytes, zeros included: a file read with fs.read, a request body, a [byte] converted with .to_str().
Levels
Level 3 is zstd's default and a good choice for most data. Levels 1 to 19 trade speed for size (19 is several times slower to compress than 3 and a little smaller); 20 to 22 use much more memory. Negative levels are faster still and compress less. A level outside zstd's range is clamped to it, and every level decompresses at the same speed.
Decompressing
unzstd reads frames whether or not they record their size: the frames the zstd command writes while streaming from a pipe, other libraries' streaming encoders, and several frames one after another (it returns their bytes joined). Input that isn't whole zstd frames gives an Err:
import compress
fn main():
print(compress.unzstd("not zstd at all"))
packed = compress.zstd("hello, hello, hello", 3)
print(compress.unzstd(packed[..len(packed) - 2]))Err("compress: unzstd: not zstd data")
Err("compress: unzstd: truncated zstd data")Untrusted input
A few bytes of zstd can expand to gigabytes. When the data comes from outside (a request body, an upload), use unzstd_max with the largest result you are willing to hold. It stops as soon as the output passes the limit, before it is all in memory:
import compress
fn main():
bomb = compress.zstd("0" * 10_000_000, 19)
print(len(bomb))
print(compress.unzstd_max(bomb, 1_000_000))326
Err("compress: unzstd: too large")Limits
A str holds up to 4GB, so that is the most either side can be. In a task, a call runs on the task's thread: a very large input holds up that core's other tasks briefly, as a big file read does. Wrap it in task.blocking if that matters.
gzip, zlib and deflate
import compress
fn main():
page = "<li>an item of a long list</li>\n" * 500
gz = compress.gzip(page, 6)
print(len(page), "->", len(gz))
print(compress.gunzip(gz) == Ok(page))
two = compress.gzip("first ", 6) + compress.gzip("second", 1)
print(compress.gunzip(two))
print(compress.gunzip("plain text"))
print(compress.gunzip_max(gz, 1000))16000 -> 107
true
Ok("first second")
Err("compress: gunzip: not gzip data")
Err("compress: gunzip: too large")Levels go from 1 (fastest) to 12 (smallest, slowest); 6 is what gzip uses by default and a good choice, and 0 stores the data uncompressed. Out of range is clamped. gunzip reads every member of a file made by joining .gz files (cat a.gz b.gz), as gzip -d does, and ignores zeros after the last one; a bad checksum, data cut short, or anything else after the end is an Err. What gzip, zlib and deflate write, other gzip and zlib programs read, and the reverse (the run tests include output from gzip -9 and Python's zlib).
Each thread keeps its compressor between calls (making one costs more than compressing a few KB), so compressing many small pieces, like HTTP responses, is cheap.
Speed
bench/std compresses ~40 MB of log lines and 2 KB pieces of them, against Go's compress/gzip and Rust's flate2 (its default backend, miniz_oxide, and zlib-rs):
| Tern | Go | Rust flate2 | Rust flate2 + zlib-rs | |
|---|---|---|---|---|
| gzip, level 1 | 623 MB/s (7.12 MB) | 327 (7.89 MB) | 491 (8.43 MB) | 536 (9.78 MB) |
| gzip, level 6 | 227 MB/s (5.55 MB) | 89 (5.65 MB) | 128 (5.65 MB) | 181 (5.94 MB) |
| gunzip | 2,936 MB/s | 623 | 963 | 1,476 |
| gzip of a 2 KB piece | 14.3 us | 30.7 | 15.9 | 18.3 |
| peak memory | 89 MB | 201 MB | 208 MB | 214 MB |
Tern's output is also the smallest at both levels.