binary
The binary module reads and writes numbers in binary formats: fixed-width integers and floats in big- or little-endian byte order, and varints (LEB128, the variable-length integers of protobuf, Go's encoding/binary, WebAssembly and SQLite records). Bytes are a str (any bytes, NULs included); a [byte] converts with to_str() and to_bytes() without a copy.
Build bytes with a Writer, read them back in order with a Reader, or read and overwrite at any offset:
import binary
import crypto
fn main():
w = binary.writer()
w.put_u16be(0xCAFE) # CA FE
w.put_u32le(7) # 07 00 00 00
w.put_uvarint(300) # AC 02
w.put_varint(-3) # 05 (zigzag: small negatives stay short)
w.put_f64be(1.5)
w.put_bytes("hi")
out = w.bytes()
print(len(out), crypto.hex(out))
r = binary.reader(out)
print(r.read_u16be(), r.read_u32le(), r.read_uvarint(), r.read_varint(), r.read_f64be(), r.read_bytes(2))
print(r.ok(), r.left())
print(binary.u32le(out, 2)) # at an offset
binary.set_u16be(&out, 0, 1) # overwrite in place
print(crypto.hex(out[..2]))19 cafe07000000ac02053ff80000000000006869
51966 7 300 -3 1.5 hi
true 0
7
0001Writer
fn writer() Writer
fn put_u8(w: mut Writer, v: int) put_u16be put_u16le put_u32be put_u32le put_u64be put_u64le
fn put_f32be(w: mut Writer, v: float) put_f32le put_f64be put_f64le
fn put_uvarint(w: mut Writer, v: int) 7 bits a byte, low first; v is read as unsigned (-1 takes 10 bytes)
fn put_varint(w: mut Writer, v: int) zigzag first: 0, -1, 1, -2 ... are 0, 1, 2, 3 ...
fn put_bytes(w: mut Writer, s: str)
fn len(w: Writer) int the bytes written so far
fn bytes(w: mut Writer) str what was written (the Writer can go on writing)
fn patch_u32be(w: mut Writer, i: int, v: int) overwrite 4 bytes written at i (a length known later)A put of an int writes its low bytes, so signed and unsigned values are written alike: put_u32be(-1) is FF FF FF FF, and put_u16le(70000) keeps the low 16 bits.
A Writer is a value like any other: copying one (w2 = w) gives two writers that go their own ways, and bytes() doesn't change when the Writer writes more afterwards.
import binary
import crypto
fn main():
a = binary.writer()
a.put_u8(1)
b = a
b.put_u8(2)
first = a.bytes()
a.put_u8(3)
print(crypto.hex(first), crypto.hex(a.bytes()), crypto.hex(b.bytes()))01 0103 0102Reader
fn reader(data: str) Reader fields: data str, pos int (where the next read starts)
fn read_u8(r: mut Reader) int read_i8 read_u16be read_u16le read_i16be read_i16le
read_u32be read_u32le read_i32be read_i32le read_u64be read_u64le
fn read_f32be(r: mut Reader) float read_f32le read_f64be read_f64le
fn read_uvarint(r: mut Reader) int read_varint
fn read_bytes(r: mut Reader, n: int) str
fn skip(r: mut Reader, n: int)
fn has(r: Reader, n: int) bool are n more bytes there?
fn left(r: Reader) int the bytes after pos
fn ok(r: Reader) bool no read has failed
fn error(r: Reader) str what the first failed read was ("" when none)A read that doesn't fit (past the end of the data, a varint cut short or longer than 64 bits, as Go's binary.Uvarint decides) returns 0 (or ""), and so does every read after it. The first problem is kept with its offset, so a parser reads a whole message and checks once:
import binary
struct Header:
kind: int
size: int
name: str
fn parse(data: str) Result[Header, str]:
r = binary.reader(data)
kind = r.read_u8()
size = r.read_u32be()
name = r.read_bytes(r.read_uvarint())
if not r.ok():
return Err(r.error())
return Ok(Header(kind, size, name))
fn main():
print(parse("\x01\x00\x00\x01\x00\x03abc"))
print(parse("\x01\x00\x00\x01\x00\x09abc"))
print(parse("\x01\x00\x00"))Ok(Header{kind: 1, size: 256, name: "abc"})
Err("binary: 9 bytes at 6 are past the end of 9")
Err("binary: 4 bytes at 1 are past the end of 3")has checks a fixed-size record once: after if not r.has(22): ... the compiler drops the checks of the reads that follow, so they cost what reads of a plain array do.
At an offset
fn u8(s: str, i: int) int i8 u16be u16le i16be i16le u32be u32le i32be i32le
fn u64be(s: str, i: int) int u64le
fn f32be(s: str, i: int) float f32le f64be f64le
fn set_u8(s: mut str, i: int, v: int) set_u16be ... set_u64le, set_f32be ... set_f64le
fn uvarint_len(v: int) int the bytes put_uvarint takes; varint_len for put_varintReading or writing past the end of s is a runtime error, as s.byte(i) is. int is 64 bits, so a u64 above 2^63-1 reads as a negative number with the same bits: u64be of eight FF bytes is -1.
Many floats at once
fn f32s_le(s: str, i: int, n: int) [f32] n f32s from byte i (a runtime error past the end)
fn f16s_le(s: str, i: int, n: int) [f32] n IEEE halves (2 bytes each) from byte i, widened to f32
fn f32s_le_bytes(xs: [f32]) str 4 bytes each
fn f16s_le_bytes(xs: [f32]) str 2 bytes each, rounded to the nearest half (inf past 65504)Vectors, embeddings and model weights are stored as raw little-endian floats (numpy's .tofile, .npy data, safetensors). These read a whole run into an [f32] in one copy, and write one back; halves save half the space where 3 significant digits are enough.
import binary
fn main():
xs: [f32] = [0.1, -2.5, 1000.0]
raw = binary.f32s_le_bytes(xs)
print(len(raw), binary.f32s_le(raw, 4, 2), binary.f32s_le(raw, 0, 3) == xs)
half = binary.f16s_le_bytes(xs)
print(len(half), binary.f16s_le(half, 0, 3))12 [-2.5, 1000.0] true
6 [0.099975586, -2.5, 1000.0]Speed and memory
bench/std does the same work with Go's encoding/binary (Append*, the byte-order getters, Uvarint) and Rust's bytes crate (BufMut on a Vec<u8>, Buf on a &[u8]) with prost's varints, the ones protobuf uses in Rust. Fixed-width: 100,000 records of a u16, a u32, a u64 and an f64 (2.2 MB) appended, then read back at offsets and in order; varints: 500,000 values of every length from 1 to 10 bytes, and 500,000 small ones (one or two bytes, as tags and lengths mostly are). Millions of values a second, best of 7 interleaved runs:
| Tern | Go | Rust | |
|---|---|---|---|
| fixed-width put | 773 | 205 | 672 |
| fixed-width get at offsets | 1343 | 727 | 1473 |
| fixed-width Reader | 1908 | 1336 | 2548 |
| varint put, 1-10 bytes | 359 | 167 | 171 |
| varint get, 1-10 bytes | 268 | 143 | 129 |
| varint put, small | 1270 | 799 | 769 |
| varint get, small | 1809 | 493 | 597 |
| peak RSS of the program | 18 MB | 46-155 MB | 22 MB |
Varints are where the formats spend their time, and there the module does no loop: up to eight bytes are gathered from (or spread into) one 64-bit word with three shift-and-mask steps (the bit moves of x86's pdep/pext, which any target can do), so lengths that vary don't cost a mispredicted branch a byte. One- and two-byte values take a test each.
A put is a compare, a store and an add: the Writer keeps its length and capacity in registers, and catches a copy of itself by keeping its buffer's own length equal to its own (a copy that wrote first has moved it). A buffer grows 2x while small and 8x past 64 KB, and an outgrown block goes back to the allocator for the next Writer to reuse.
Rust leads the two fixed-width read rows. Its loop moves one slice pointer and reads at constant offsets from it; Tern's reads are the same loads with no checks left, but the C compiler gives each its own base pointer, and in a large function those spill.