# Tern: the complete reference for LLMs Generated by tools/llms.tn from docs/reference.md, docs/grammar.md, library/std/*.tn, and example programs. Tern source files end in .tn. Check a program with `tern check --json prog.tn` and run it with `tern run prog.tn`. Programs marked `(rejected on purpose)` show mistakes the compiler refuses. # Tern language reference This is the complete reference for writing Tern programs. It is written so that a reader who has never seen Tern (a person or an LLM) can write correct code on the first try. Every ` ```tern ` block below is a complete program that the test suite (`tern run tests/runner/main.tn -- docs`) compiles and runs; when an ` ```output ` block follows, the program's output must match it exactly. So the examples here are known to work. ## 0. The short version Tern looks like Python, but it is statically typed and compiled, and **every value is a copy**: there are no references, pointers, `None`, classes, or exceptions. The rules that differ most from Python: 1. Functions are `fn name(a: int, b: str) int:`. Every parameter needs a type. The return type comes after the `)` with no arrow (`-> int` is also accepted). No return type = returns nothing. 2. `x = 5` creates a variable. Its type is fixed from then on. `x: float = 0.0` states a type. There is no `let` or `var`, and there are no global variables. Top-level `const NAME = value` declares a constant (computed at compile time, or on first use when it calls a function). 3. The top level of a file may only contain `import`, `from ... import`, `struct`, `enum`, `const`, `fn`, `extern`, and `export fn`. The program starts at `fn main():`. 4. Types: `int` (64-bit), `float` (64-bit), `bool`, `str`, `[T]` (array), `{K: V}` (map), structs, enums, tuples `(int, str)`, `fn(int) str` (functions). `==` compares any of them (except functions) by value. Generics: `fn first[T](xs: [T]) T`, `struct Pair[A, B]`. No null: a value that may be missing is an `Option[T]` (`Some(x)` or `None`), a result that may fail a `Result[T, E]` (§9). 5. `true`, `false` (lowercase). Logic is `and`, `or`, `not`. Strings use double quotes only. 6. No implicit conversions: `1 + 2.0` is an error. Write `float(1) + 2.0` or `1 + int(2.0)`. `str + int` is an error: write `"n=" + str(n)` or `f"n={n}"`. 7. `/` on two ints is integer division (truncates toward zero). There is no `//`. 8. Parameters are read-only. To let a function change the caller's variable, declare the parameter `mut` and pass it with `&`: `fn inc(n: mut int)` is called as `inc(&x)`. 9. Assignment copies: after `b = a`, changing `b` never changes `a` (this includes arrays, maps, and structs; the copy is lazy, so it is cheap until one side is written). 10. `for i in 0..n:` loops over `0, 1, ..., n-1`. There is no `range()`. 11. Arrays: `xs.push(v)`, `xs.pop()`, `len(xs)`. `push` copies `v` into the array, like `=` (so `v` stays usable, and it moves for free when `v` isn't used again). There is no `append`, list comprehension, or negative index. Use `xs.sorted()`, `xs.reversed()`, `sum(xs)`, `xs.sort_by(fn(x) key)`, `xs.map(...)`, `xs.filter(...)`: each returns a new value and leaves `xs` alone. 12. Structs have no methods inside them. Write a function whose first parameter is the struct; then `p.area()` calls `area(p)`. 13. Your functions may share a builtin's name (`fn get(s: Store, id: int)`): a call uses yours when its first parameter fits the first argument, and the builtin otherwise. See §20. A complete program: ```tern struct Item: name: str price: float qty: int fn total(items: [Item]) float: t = 0.0 for it in items: t += it.price * float(it.qty) return t fn main(): items: [Item] = [] items.push(Item("apple", 0.5, 4)) items.push(Item(name="pear", price=0.75, qty=2)) print(f"{len(items)} items, total {total(items)}") for i, it in items: print(i, it.name) ``` ```output 2 items, total 3.5 0 apple 1 pear ``` ## 1. Running programs ```text tern run prog.tn compile and run (needs clang) tern run prog.tn -- a b pass arguments (read them with os.args()) tern check prog.tn type/safety check only; prints "prog.tn: ok" tern check --json prog.tn errors as JSON (for tools and agents) tern test prog_test.tn run the `test_*` functions (see the `testing` module, §16) tern fmt prog.tn reformat the file in place tern build prog.tn -o prog make a native binary tern emit prog.tn print the generated C tern guide print this reference tern guide strings print only the sections about a topic ``` `check --json` prints `{"ok": true, "file": ...}` or `{"ok": false, "errors": [...]}`, where each error is `{"file", "line", "message", "hint", "source"}` (`hint` is `null` when there is none; `source` is the offending line). Compile errors look like `prog.tn:12: error: `, often followed by a `hint:` line when the code looks like another language. All errors in a file are reported at once. A runtime error (index out of bounds, missing map key, division by zero, `pop()` on an empty array) prints `runtime error in f (prog.tn:N): ` to stderr (`(util.tn, line N)` when it happened in another file), then the Tern functions it came through (` in util.helper <- run <- main`), and exits with status 2. ## 2. Lexical rules - Comments start with `#` and run to the end of the line. There are no block comments or docstrings. - Blocks are made by indentation after a `:`, like Python. Use 4 spaces. A block with one statement may go on the same line: `if x > 0: print(x)`. - A line continues while a `(`, `[`, or `{` is open, or after a trailing `\`. - No semicolons. - Integers: `42`, `1_000_000`, hex `0xff`, binary `0b1010`, octal `0o17` (`_` may separate digits anywhere). A decimal literal must fit in an int (up to `9223372036854775807`); a hex, binary, or octal one may use all 64 bits, and above that value it is negative (`0xffff_ffff_ffff_ffff` is `-1`), which is handy for hash constants and masks. A literal that doesn't fit is a compile error. - Floats: `1.0`, `0.5`, `1e9`, `2.5e-3`, `6.02E+23` (an exponent makes a float). A dot needs digits on both sides: not `1.` or `.5`. - Strings: `"double quotes"`, on one line. Escapes: `\n \t \r \0 \" \\`; `\xNN` is one byte (`"\x01"`, `"\xff"`); `\u00e9` (four hex digits) and `\u{1F600}` (one to six) are a Unicode character, stored as UTF-8. A malformed `\x` or `\u` is a compile error. Any other backslash is kept as is: `"a\qb"` is the four bytes `a \ q b`. - Multi-line strings: `"""` ... `"""` (and `f"""` ... `"""`) may span lines and take the same escapes; `"` needs no escape inside. When the text starts on the line after the opening `"""`, that line break and the one before the closing `"""` are dropped, and the indentation all its lines share is removed, so a block of SQL or HTML can be indented with the code. - Raw strings: `` `C:\dir\file` `` between backticks: every character as written, no escapes, may span lines (a backtick can't appear inside). Good for regular expressions and JSON. - F-strings: `f"x = {x}, next = {x + 1}"`. Any expression whose value is an `int`, `float`, `bool`, or `str` goes in `{}` (for an array, map, or struct, use `print(label, xs)` or `json.encode(xs)`); `{{` and `}}` are literal braces. No format specs (`{x:.2f}` is an error): use the `fmt` module, `f"{fmt.fixed(x, 2)}"` (§12). - Names: letters, digits, `_`. Names starting with `_` are private to their module. - Keywords: `and break const continue do elif else enum export extern false fn for from if import in mut not null or parallel pass return struct take true while`. `match` is a soft keyword, `type` is one at the top level (`type Name = T`), and `is` is one after an expression (`x is None`). ## 3. Variables and types ```tern fn main(): n = 10 # int ratio = 0.25 # float ok = true # bool name = "tern" # str xs = [1, 2, 3] # [int] ages = {"ada": 36} # {str: int} empty: [str] = [] # an empty collection needs a type... later = [] # ...or gets it from its first use later.push(1.5) n = n + 1 # reassigning keeps the type print(n, ratio, ok, name, xs, ages, empty, later) ``` ```output 11 0.25 true tern [1, 2, 3] {"ada": 36} [] [1.5] ``` - A variable is created by its first assignment and lives until the end of the block it was created in. Create it before an `if` if you need it after: ```tern fn main(): label = "" if 3 > 2: label = "big" else: label = "small" print(label) ``` ```output big ``` - There is no declaration without a value (`x: int` alone is an error); use `x = 0`. - Constants go at the top level: `const NAME = value` or `const NAME: type = value`. A value made of literals, other such constants, and operators (including `a if c else b`) is an int, float, str, or bool computed when compiling; it can also be a `match` pattern. A constant can be used anywhere a value can, and from other modules (`config.PORT`, `from config import PORT`). There are no global variables: constants can't change, so every task can read them. - Any other value (a function call, an array, a struct, a compiled regex, a `task.shared_map()` cache) is computed when the program first uses it, once, even if many tasks use it at the same moment, and kept: `const EMAIL = re.must("[^@ ]+@[^@ ]+")` compiles the pattern once for the whole program. Its type comes from the value (write it, `const X: [str] = []`, when it can't). It can't be a `match` pattern, and a value that needs itself is a runtime error. ```tern const MAX_USERS = 100 const NAME = "tern" const BANNER = NAME + " v2" const MASK: int = (1 << 12) - 1 const LIMIT = MAX_USERS * 2 if MASK > 0 else 0 fn kind(n: int) str: match n: MAX_USERS: return "full" _: return "ok" fn main(): print(BANNER, LIMIT, MASK, kind(100), kind(7)) ``` ```output tern v2 200 4095 full ok ``` ```tern import re import task const WORD = re.must("[a-z]+") const STOP = ["the", "a", "of"] const SEEN = task.shared_map() # one cache for the whole program fn words(s: str) int: n = 0 for m in WORD.find_all(s): if not (m.text in STOP): SEEN.add_int(m.text, 1) n += 1 return n fn main(): lines = ["the cat sat", "a cat of note"] parallel for i in 0..len(lines): words(lines[i]) print(SEEN.get_int("cat"), SEEN.get_int("the"), len(STOP)) ``` ```output 2 0 3 ``` - Types can't change: `x = 1` then `x = "a"` is an error. - Type syntax: `int float bool str byte f32`, `bytes16` and `f32x4` (16 bytes / four f32s at once: std simd), `[int]`, `[[str]]`, `{str: int}`, `{int: [Point]}`, `(int, str)`, `Point`, `http.Request` (a type from a module), `fn(int, str) bool`, `fn(str)` (returns nothing), `ptr` (C pointer). - Wrong type names from other languages are errors: `string`, `i32`, `i64`, `f64`, `list[int]`, `List`, `dict`, `Vec`. (`void` is accepted only as a return type, where it means the same as leaving the return type out.) - Default values when you need one: `0`, `0.0`, `false`, `""`, `[]`, `{}`. ## 4. Operators | Operator | Works on | Notes | |---|---|---| | `+ - * /` | int with int, float with float | `/` on ints truncates toward zero: `7 / 2 == 3`, `-7 / 2 == -3` | | `%` | int | sign follows the left side: `-7 % 3 == -1` | | `+` | str with str | concatenation | | `*` | str with int | repetition: `"ab" * 3` | | `== !=` | any two values of the same type | structs, tuples, and enums field by field, arrays and maps element by element; not functions or task handles | | `< <= > >=` | int, float, str | strings compare by bytes | | `and or not` | bool | short-circuit | | `in` | `x in xs` (array), `k in m` (map key), `"a" in s` (substring) | | | `& \| ^` | int with int | bitwise and, or, xor (for bools, use `and` / `or`) | | `<< >> >>>` | int with int | shifts: `>>` keeps the sign (arithmetic), `>>>` shifts in zeros (logical) | | `+= -= *= /= %=` | same as the operator | also `&= \|= ^= <<= >>= >>>=`; no `++` or `--` | | `-x` | int, float | | | `~x` | int | bitwise not: `~x == -x - 1` | | `&x` | a variable, field, or element | only as an argument to a `mut` parameter | | `x is P` | an enum value | true if `x` is variant `P`: `o is None`, `r is Ok(_)`, `if o is Some(v):` (§9) | | `x?` | an Option or a Result | the value in `Some` / `Ok`, else return the `None` / `Err` at once (§9) | - Int arithmetic wraps on overflow (64-bit two's complement). Division or `%` by zero is a runtime error. - A shift count must be 0 to 63: `1 << 64` is a compile error when the count is written as a number, and a runtime error otherwise. `<<` shifts bits out of the top (it wraps like `*`). - Precedence, lowest first: `or`, `and`, comparisons and `in`, `|`, `^`, `&`, shifts, `+ -`, `* / %`, then the unary `-x`, `~x`, and `not x` (so write `not (a == b)`). As in Python (not C), `x & 4 == 4` means `(x & 4) == 4`. ```tern fn fnv1a(s: str) int: # a 64-bit hash h = 0xcbf2_9ce4_8422_2325 for c in s: h ^= c.byte(0) h *= 0x100_0000_01b3 return h fn main(): flags = 0b0101 flags |= 1 << 3 # set bit 3 print(flags, flags & 0b100 != 0, flags >> 1, ~0) print(-16 >> 2, -16 >>> 60, 0xff, 0o17, 1e3) print(fnv1a("tern")) ``` ```output 13 true 6 -1 -4 15 255 15 1000.0 -440267618092719894 ``` - `a if cond else b` is a conditional expression (as in Python; there is no `c ? a : b`). Only the chosen side is evaluated, both sides have the same type, and it binds loosest of all, so `x = n * 2 if n > 0 else 0` means `x = (n * 2) if n > 0 else 0`: ```tern fn sign(n: int) str: return "negative" if n < 0 else "zero" if n == 0 else "positive" fn main(): xs = [3, -1, 0] print(xs.map(fn(x) sign(x)), [] if len(xs) > 5 else xs[..1]) ``` ```output ["positive", "negative", "zero"] [3] ``` - There is no `**` (use `math.pow`) or `//`. - Mixing int and float is always an error; convert with `float(i)` or `int(f)` (truncates toward zero; NaN, infinities, and floats outside int's range are a runtime error). ## 5. Control flow ```tern fn main(): for i in 0..3: # 0, 1, 2 (end is exclusive; no step) print("i", i) xs = ["a", "b"] for x in xs: # elements print(x) for i, x in xs: # index and element print(i, x) m = {"one": 1, "two": 2} for k, v in m: # key and value, in insertion order print(k, v) n = 3 while n > 0: # counting down: use while n -= 1 if n == 1: continue print("n", n) x = 7 if x < 5: print("small") elif x < 10: # `elif`, never `else if` print("medium") else: print("large") for i in 0..100: if i == 2: break if x > 0: pass # an empty block needs `pass` ``` ```output i 0 i 1 i 2 a b 0 a 1 b one 1 two 2 n 2 n 0 medium ``` - `for i in a..b` with `a >= b` runs zero times. To count down, use `while`. - The loop variable can't be assigned, and the array or map being looped over can't be changed inside the loop. To change elements while looping, loop over indices: `for i in 0..len(xs): xs[i] = xs[i] * 2`. - `match` is covered in §9. There is no `switch`, `try`, `with`, `goto`, or `do/while`. ## 6. Functions ```tern fn add(a: int, b: int) int: return a + b fn greet(name: str): # no return type: returns nothing print("hello " + name) fn divmod(a: int, b: int, q: mut int, r: mut int): q = a / b # `mut` parameters write back to the caller r = a % b fn main(): greet("ada") q = 0 r = 0 divmod(17, 5, &q, &r) # `&` marks each argument that may change print(add(2, 3), q, r) ``` ```output hello ada 5 3 2 ``` - Every parameter has a type. There are no default values, keyword arguments (named arguments are only for building structs), variadic parameters, or overloading. - A function with a return type must end with a `return` on every path. - Parameter modes: - default: read-only. Assigning to it, or calling `push` on it, is an error. - `mut`: the function may change it and the caller sees the change. The caller must write `&x`. - `take`: the value moves into the function (no copy). The caller can't use `x` afterwards (pass `copy(x)` to keep a copy). - To return several values, return a tuple: `fn divmod(a: int, b: int) (int, int):` then `return q, r`, and the caller writes `q, r = divmod(7, 2)` (§7, Tuples). - To report failure, return a `Result[T, E]` (`Ok(value)` or `Err(error)`), or an `Option[T]` when there's nothing to say about why (§9). There are no exceptions. - Method syntax: `x.f(a, b)` is the same call as `f(x, a, b)`. If the first parameter is `mut`, write `x.f(...)` or `f(&x, ...)`. `x.f` also finds `f` in the module where `x`'s type is defined, so `d.exec(...)` calls `sqlite.exec(d, ...)`. - Functions can be defined inside functions (they are closures, §13) and may recurse. Top-level functions may be called before they are defined. ## 7. Structs ```tern struct Point: x, y: int # several fields of one type struct Player: name: str pos: Point scores: [int] fn dist(p: Point) int: # "methods" are plain functions return abs(p.x) + abs(p.y) fn move(p: mut Player, dx: int): # a "mutating method" p.pos.x += dx fn main(): a = Point(1, 2) # positional: fields in order b = Point(y=5, x=-3) # named: any order; every field is required pl = Player("zed", a, []) pl.scores.push(10) # nested fields and arrays change in place pl.move(4) # same as move(&pl, 4) c = a # a copy c.x = 100 print(a.x, c.x, b.dist(), pl) ``` ```output 1 100 8 Player{name: "zed", pos: Point{x: 5, y: 2}, scores: [10]} ``` - All fields must be given when building a struct; there are no default field values. Write a constructor function if you want defaults: `fn new_player(name: str) Player:`. - No `self`, no classes, no inheritance, no interfaces. `==` compares two structs field by field; when equality means something else, write `fn same_user(a: User, b: User) bool`. - A struct may contain an array of itself (`kids: [Node]`) but not itself directly. For graphs, store nodes in an array and refer to them by index. - `print` shows structs as `Name{field: value, ...}`. ### Tuples A tuple is a fixed group of values of any types, written `(a, b)`; its type is written the same way, `(int, str)`. It is a value like a struct whose fields have no names: read them by position with `t.0`, `t.1`. Functions return several values as a tuple, and `a, b = ...` unpacks one (each name on the left is a new variable, an existing one, a field or element, or `_` to skip a value). ```tern fn divmod(a: int, b: int) (int, int): return a / b, a % b # parentheses are optional after `return` and `=` fn main(): q, r = divmod(17, 5) print(q, r) t = divmod(9, 2) print(t, t.0, t.1) a = 1 b = 2 a, b = b, a # swap _, rest = divmod(7, 3) scores: [(str, int)] = [("ada", 3), ("bob", 5)] scores.push(("cy", 4)) for s in scores: print(s.0, s.1) print(a, b, rest, scores[1]) ``` ```output 3 2 (4, 1) 4 1 ada 3 bob 5 cy 4 2 1 1 ("bob", 5) ``` - A tuple has two or more values; there is no one-value tuple. `t.0 = 5` changes one value. - The number of names on the left of `a, b = ...` must match the tuple. - Tuples can go anywhere a type can: in arrays, maps, struct fields, and function types. - `print` shows `(1, "a")`, and `==` compares tuples value by value. Tuples can't be turned into JSON yet. ### Generics Functions, structs, and enums can take type parameters in `[ ]` after their name. A use with concrete types makes a version of the function or type for those types (like C++ templates or Rust): it runs exactly as fast as code written by hand for them. ```tern struct Pair[A, B]: left: A right: B enum Tree[T]: # recursive generic enums work too Leaf Node(l: Tree[T], v: T, r: Tree[T]) fn first[T](xs: [T]) T: return xs[0] fn largest[T: Ord](xs: [T]) T: # a constraint: T must support < and > best = xs[0] for x in xs: if x > best: best = x return best fn map_all[A, B](xs: [A], f: fn(A) B) [B]: out: [B] = [] for x in xs: out.push(f(x)) return out fn insert[T: Ord](t: Tree[T], v: T) Tree[T]: match t: Leaf: return Tree.Node(Tree[T].Leaf, v, Tree[T].Leaf) Node(l, x, r): if v < x: return Tree.Node(insert(l, v), x, r) return Tree.Node(l, x, insert(r, v)) fn main(): print(first([3, 1]), first(["a"]), largest([2.5, 9.0])) p = Pair(1, "one") # Pair[int, str], from the values q: Pair[str, [int]] = Pair("n", []) # the declared type fills in [int] print(p, q, map_all([1, 2], fn(x) x * 10), first[str](["z"])) t = Tree[int].Leaf # explicit type arguments for v in [5, 2, 8]: t = insert(t, v) print(t) ``` ```output 3 a 9.0 Pair{left: 1, right: "one"} Pair{left: "n", right: []} [10, 20] z Tree.Node(l: Tree.Node(l: Tree.Leaf, v: 2, r: Tree.Leaf), v: 5, r: Tree.Node(l: Tree.Leaf, v: 8, r: Tree.Leaf)) ``` - Type arguments are usually inferred: from the arguments, then from the type the result is going into (a typed variable, a `return`, a parameter), then from lambdas. When they can't be, write them: `first[int](xs)`, `Pair[int, str](1, "a")`, `Tree[int].Leaf`. - A generic type always needs its arguments where a type is written: `Pair[int, str]`, not `Pair`. - Constraints: `T: Ord` (int, float, str: `<`, `>`, sorting), `T: Eq` (`==`), `T: Hash` (usable as a map key); several: `T: Ord + Hash`. Without a constraint any type is accepted, and the body is checked for each type it is used with; an error there names the version, e.g. ``(in `total[Point]`, used on line 12)``. - A generic function used as a value needs its arguments: `f = first[int]`. - There are no traits or interfaces: pass behavior as a function value (`fn(T, T) bool`). ### Type aliases `type Name = T` at the top level of a module gives a type a second name. The alias *is* the type: values pass freely between the two names, `Name(...)` constructs one, and `x.f()` finds `f` in the module that defines the original. A package uses aliases to present types that live elsewhere under its own name (`roost.Request` is `http.Request`). ```tern struct Point: x: int y: int type Pos = Point type Path = [Pos] type Scores = {str: int} fn walk(p: Path) int: total = 0 for q in p: total += abs(q.x) + abs(q.y) return total fn main(): a = Pos(3, -4) b: Point = a # the same type under two names s: Scores = {"ada": walk([a, b])} print(a, s) ``` ```output Point{x: 3, y: -4} {"ada": 14} ``` - The alias's target is resolved in the alias's own module, so `type Request = http.Request` needs `import http` there; users of the alias don't. - An alias can name any type (`[T]`, `{K: V}`, `fn(int) str`, another alias), but not itself, directly or through other aliases. - Errors and printed values use the original name (`Point` above). - `type` is a soft keyword: only `type NAME =` at the top level declares an alias, so `type` still works as a field or variable name. ## 8. Arrays `[T]` ```tern fn main(): xs = [3, 1, 2] xs.push(5) # append at the end last = xs.pop() # remove and return the last element print(xs, last, len(xs), xs[0]) xs[0] = 30 zeros = [0] * 4 # [value] * n grid = [[0] * 3] * 2 # 2 rows of 3 (each row is its own copy) grid[1][2] = 7 print(zeros, grid) print(xs[1..], xs[..2], xs[1..2], 2 in xs) # slices are copies ys = xs.sort_by(fn(x) x) # returns a new sorted array (stable) down = xs.sort_by(fn(x) -x) # descending: negate a number key evens = [1, 2, 3, 4].filter(fn(x) x % 2 == 0) squares = [1, 2, 3].map(fn(x) x * x) print(ys, down, evens, squares, xs) print(xs.sorted(), xs.reversed(), sum(xs), sum([0.5, 0.25]), max(xs), xs.min()) words = ["pear", "fig"] print(words.sorted(), words.sort_by(fn(w) len(w))) ``` ```output [3, 1, 2] 5 3 3 [0, 0, 0, 0] [[0, 0, 0], [0, 0, 7]] [1, 2] [30, 1] [1] true [1, 2, 30] [30, 2, 1] [2, 4] [1, 4, 9] [30, 1, 2] [1, 2, 30] [2, 1, 30] 33 0.75 30 1 ["fig", "pear"] ["fig", "pear"] ``` - `xs.extend(ys)` pushes every element of `ys` onto `xs`. - `xs.reserve(n)` makes room for `n` more elements in one block, so pushing that many doesn't grow and copy the array on the way (like Go's `make([]T, 0, n)` or Rust's `Vec::with_capacity`). It changes nothing else. - `xs.resize_uninit(n)` makes an `[int]`, `[float]`, `[f32]` or `[byte]` exactly `n` long without writing the new elements: their contents are undefined (whatever the memory held, possibly another array's old data) until you write them. It is for big arrays that are about to be filled completely, e.g. by C code, where `[0] * n` would cost a pass over the memory first. Other element types start with values (`[""] * n`). - Slices clamp to the length (`[1, 2, 3][1..10]` is `[2, 3]`), for strings too. - Indexes start at 0. Out-of-range indexes are runtime errors. There are no negative indexes: the last element is `xs[len(xs) - 1]`. - `xs.sorted()` is a sorted copy of an array of ints, floats, or strs; `xs.reversed()` is a copy in reverse order; `sum(xs)` adds an `[int]` or `[float]` (0 when empty; int sums wrap like `+`), and `min(xs)`, `max(xs)` find its smallest and largest (an empty array is a runtime error). None of them changes `xs`: to sort in place, write `xs = xs.sorted()`. - `sort_by(xs, key)` sorts by a key that is an `int`, `float`, or `str` (stable). It never changes `xs`; it returns a new array. Descending: `xs.sort_by(fn(x) -x)` for numbers, `xs.sorted().reversed()` for strings. - Other operations you might expect are short loops: `index_of`, `insert`, `zip`. Example: `for i in 0..len(xs): if xs[i] == v: return i`. - Removing from the middle: build a new array with `filter`, or with a loop. - `push` copies its value into the array the way `=` does: lazily, and not at all when the pushed variable isn't used afterwards (then the value moves). ```tern fn main(): names: [str] = [] name = "ada" names.push(name) # `name` is still usable name += "!" # and changing it doesn't change the array names.push("bob") print(name, names) ``` ```output ada! ["ada", "bob"] ``` - `copy(x)` makes an explicit copy. It is only needed for `take` parameters (§6). ### Bytes `byte` is a number from 0 to 255, for binary data: pixels, network buffers, hashes. A `[byte]` stores one byte per element (an `[int]` uses eight), so a 1024x1024 RGBA image is 4MB. ```tern fn main(): px: [byte] = [255, 128, 0, 64] # a number 0-255 is a byte where one is expected gray = (px[0] * 77 + px[1] * 150 + px[2] * 29) >> 8 # bytes in arithmetic are ints out = [byte(0)] * 3 out[0] = byte(gray) # storing an int takes byte(n): its low 8 bits out[1] = byte(300) out[2] = 250 out[2] += 10 # op= on a byte wraps data = "Tern".to_bytes() # str <-> [byte] share memory: no copy data[0] = 116 print(gray, out, data.to_str(), int(px[1]) * 2) ``` ```output 151 [151, 44, 4] tern 256 ``` - In arithmetic and comparisons a byte counts as an `int`, and so does the result: `px[0] * 77` can't overflow. `-b` is an int too. - Storing an int into a byte (a variable, element, field, or argument) takes `byte(n)`, which keeps the low 8 bits. A literal from 0 to 255 needs no conversion; one outside that range is an error. `b += n` (any `op=`) on a byte wraps, as `byte(b + n)` would. - `int(b)` and `float(b)` convert back. `s.byte(i)` still returns an `int`. - `s.to_bytes()` gives a string's bytes, `bs.to_str()` makes a string of them. Both share the block (a `str` and a `[byte]` are laid out the same); writing to either copies it first. - `print` and `json.encode` show bytes as numbers; `json.decode` rejects numbers outside 0-255. ### 32-bit floats `f32` is a 32-bit float, for big arrays of numbers: vectors, embeddings, audio, model weights. An `[f32]` stores 4 bytes per element (a `[float]` uses eight), so a million 768-number embeddings take 3GB instead of 6GB. `std simd` works on four of them at once (`f32x4`), and `std binary` reads and writes them as raw bytes (`binary.f32s_le`, `binary.f16s_le`). ```tern fn dot(a: [f32], b: [f32]) f32: s: f32 = 0.0 # a float literal is an f32 where one is expected for i in 0..len(a): s += a[i] * b[i] # f32 * f32 is an f32 return s fn main(): xs: [f32] = [0.5, 1.5, -2.0] ys = [f32(1), f32(2), 0.25 * xs[0]] # f32(n) converts an int or a float d = dot(xs, ys) print(d, float(d) * 2.0, xs.sorted(), sum(xs), f32(0.1), float(f32(0.1))) ``` ```output 3.25 6.5 [-2.0, 0.5, 1.5] 0.0 0.1 0.10000000149011612 ``` - Math on f32s stays in f32, as in Rust, Go and C. An f32 and a float don't mix: write `f32(x)` or `float(x)` (a literal like `0.5` adapts by itself). - `print`, `str` and f-strings show the shortest decimal that reads back as the same f32 (`0.1`, not `0.10000000149011612`); `json.encode` writes the same, and `json.decode` rounds. - `int(x)`, `abs`, `sqrt`, `min`, `max`, `sum`, `sorted` and `sort_by` keys take f32s. An f32 can't be a map key (nor can a float). - In an `extern` block an f32 is a C `float`, and an `[f32]` a `float*`. ## 9. Enums and `match` ```tern import math enum Shape: Circle(radius: float) Rect(w: float, h: float) Empty enum Expr: # recursive enums just work Num(v: int) Add(l: Expr, r: Expr) fn area(s: Shape) float: match s: Circle(r): # bind payload values by position return math.pi() * r * r Rect(w, h): return w * h Empty: return 0.0 fn eval(e: Expr) int: match e: Num(v): return v Add(l, r): return eval(l) + eval(r) fn describe(code: int) str: match code: # match on int, str, or bool literals 200: return "ok" 404: return "not found" 301 | 302 | 307: # several patterns, one arm return "redirect" _: # `_` matches anything else return "other" fn main(): shapes = [Shape.Circle(1.0), Shape.Rect(w=2.0, h=3.0), Shape.Empty] for s in shapes: print(area(s)) print(eval(Expr.Add(Expr.Num(2), Expr.Num(3))), describe(404)) print(shapes[2]) ``` ```output 3.141592653589793 6.0 0.0 5 not found Shape.Empty ``` - Build variants as `Enum.Variant(values)` or `Enum.Variant`. In patterns, write the bare variant name (`Circle(r)`) or `Shape.Circle(r)`; `_` ignores one value (`Rect(w, _)`). An enum from another module is named through it: `json.Value.Null`, or just `json.Null` when only one of the module's enums has that variant (both in patterns too). - A `match` must handle every value: on an enum, every variant (or end with `_:`; missing variants are listed in the error); on an int or str, it needs a `_:` arm (`_: pass` when nothing should happen); on a bool, `true` and `false` or `_:`. - Enum payloads can only be read with `match` (no `s.radius`). Enums without payloads can be compared with `==`. - Each arm is a block; there is no fallthrough and no `case` keyword, and no guards (`if` inside arm bodies instead). Patterns that share a body are joined with `|`: `"cat" | "dog":`, `Dot | Point:` (variants that bind names each need their own arm). - `x is Pattern` tests which variant `x` is: `if s is Circle(r):` (see Option below). ### Option and Result Two generic enums are always available, without an import: ```text enum Option[T]: enum Result[T, E]: Some(value: T) Ok(value: T) None Err(error: E) ``` `Option[T]` is a value that may be missing (Tern has no null); `Result[T, E]` is a result or an error (Tern has no exceptions). Build them bare: `Some(3)`, `None`, `Ok(v)`, `Err("bad")`. Test them with `match`, or with `x is Pattern`, whose names are bound in the `if` or `while` block. And `x?` gives the value inside `Some` / `Ok`, or else returns the `None` / `Err` from the function at once (the function must return an Option, or a Result with the same error type). ```tern struct User: name: str age: int fn find_user(us: [User], name: str) Option[User]: for u in us: if u.name == name: return Some(u) return None fn parse_age(s: str) Result[int, str]: n = s.to_int(-1) if n < 0: return Err(f"bad age: {s}") return Ok(n) fn total_age(texts: [str]) Result[int, str]: t = 0 for s in texts: t += parse_age(s)? # an Err returns from total_age right here return Ok(t) fn main(): us = [User("ada", 36), User("bob", 25)] if find_user(us, "bob") is Some(u) and u.age > 20: print(u.name, u.age) match find_user(us, "zed"): Some(u): print("found", u.name) None: print("no zed") print(total_age(["1", "2"]), total_age(["1", "x"])) print(parse_age("7").unwrap_or(0), parse_age("?").unwrap_or(0), find_user(us, "q").is_none()) cur: Option[int] = Some(3) while cur is Some(n): # a loop that runs while there is a value cur = Some(n - 1) if n > 1 else None print(cur) ``` ```output bob 25 no zed Ok(3) Err("bad age: x") 7 0 true None ``` - Methods: `o.unwrap_or(default)`, `o.unwrap()` (a runtime error on None), `o.expect(msg)`, `o.is_some()`, `o.is_none()`; on a Result: `r.unwrap_or(default)`, `r.unwrap()`, `r.expect(msg)`, `r.is_ok()`, `r.is_err()`, `r.ok()` (an Option of the value), `r.err()`. - `None` needs to know its type: from a typed variable (`x: Option[int] = None`), a return type, or a parameter. Otherwise write `Option[int].None`. - `x is V(a, b)` binds `a` and `b` only as the condition of an `if` / `while`, or on the left of `and` there (`if o is Some(v) and v > 0:`). Anywhere else it's a plain test: `x is None`, `r is Ok(_)`. There's no `is not`: write `not (x is None)`. - `is` works on every enum, not just these two. A module can define its own `Option`, `Some`, and so on; its own names win. - `r.context("loading config")` turns an `Err(e)` into `Err("loading config: " + e)` (for a `Result[T, str]`), `r.map_err(f)` transforms the error, and `o.ok_or(err)` turns an Option into a Result. - `panic(msg)` stops with a runtime error on purpose (for "can't happen"), and `assert(cond, msg)` does when `cond` is false; the message is built only then. Like any runtime error, a `panic` in a task is raised again by `h.join()`, and one in a web handler answers 500. A function ending in `panic(...)` needs no `return` after it. - `main` may return a `Result[int, str]`: `Err(msg)` prints `error: msg` and exits with status 1, `Ok(n)` exits with status `n`. So `?` works in `main` too: ```tern import fs fn read_port(path: str) Result[int, str]: text = fs.try_read(path).context("reading " + path)? n = text.trim().to_int(-1) assert(n != 0, "port 0 is reserved") if n < 0: return Err(f"not a port: {text.trim()}") return Ok(n) fn main() Result[int, str]: fs.write("/tmp/tern_port.txt", "8080\n") port = read_port("/tmp/tern_port.txt")? print("port", port, read_port("/tmp/missing.txt").is_err()) return Ok(0) ``` ```output port 8080 true ``` ## 10. Strings `str` Strings are UTF-8 bytes. `len`, indexing, and slicing count **bytes**, and `s[i]` is a one-byte `str`. For ASCII text this is the same as characters. ```tern fn main(): s = "Hello, World" print(len(s), s[0], s[7..], s[..5], s[7..9]) print(s.find("o"), s.find("zz"), s.contains("World"), "World" in s) print(s.starts_with("He"), s.ends_with("!"), s.upper(), s.lower()) print(s.replace("l", "L"), " pad ".trim(), "ab".repeat(3), "ab" * 2) parts = "a,b,,c".split(",") print(parts, len(parts), "-".join(parts)) print("42".to_int(0), "x".to_int(-1), "2.5".to_float(0.0), str(42) + str(2.5) + str(true)) print(s.byte(0), chr(72), "b" > "a") out = "" for ch in "abc": # one byte at a time out = ch + out print(out, f"{s.lower()}: {len(s) * 2}") ``` ```output 12 H World Hello Wo 4 -1 true true true false HELLO, WORLD hello, world HeLLo, WorLd pad ababab abab ["a", "b", "", "c"] 4 a-b--c 42 -1 2.5 422.5true 72 H true cba hello, world: 24 ``` | Function | Result | |---|---| | `len(s)` / `s.len()` | length in bytes | | `s[i]`, `s[a..b]`, `s[a..]`, `s[..b]` | one-byte string, substring | | `s.find(sub)` | index of the first match, or `-1` | | `s.find_from(sub, start)` | the first match at or after `start`, or `-1` | | `s.strip_prefix(p)`, `s.strip_suffix(p)` | `s` without `p` at its start / end (`s` itself if it isn't there) | | `s.contains(sub)`, `sub in s` | bool | | `s.starts_with(p)`, `s.ends_with(p)` | bool | | `s.replace(old, new)` | every occurrence replaced | | `s.trim()` | without leading/trailing whitespace | | `s.upper()`, `s.lower()` | ASCII case change | | `s.split(sep)` | `[str]` (empty pieces kept) | | `sep.join(parts)` | `str` (called on the separator, like Python) | | `s.to_int(default)`, `s.to_float(default)` | parsed number, or `default` if `s` isn't one (`to_int` doesn't allow spaces, and a number outside the int range gives `default`; `to_float` allows leading spaces) | | `s.byte(i)` | the byte at `i` as an `int` | | `s.reserve(n)` | room for `n` more bytes (`s` is changed: a `mut` place), so building a long string by `+=` doesn't grow and copy it on the way (`xs.reserve(n)` does the same for an array: `n` more elements) | | `s.repeat(n)`, `s * n` | repeated | | `str(x)` | `x` must be int, float, bool, or str | | `chr(n)` | one-byte string from a byte value (0 to 255; anything else is a runtime error) | - Strings are immutable values: `s[0] = "x"` is an error; build a new string instead. - `int("42")` is an error (`int()` converts floats). Use `"42".to_int(0)`. - `str()` of an array or struct is an error; `print` shows any value, and `json.encode` makes text from any value. - Building a big string in a loop with `+=` is fine (it grows in place). ### Multi-line and raw strings ```tern fn main(): user = "ada" sql = """ SELECT id, name FROM users WHERE name = $1 """ page = f"""

Hello, {user}

{len(sql)} bytes of SQL

""" pattern = `^\d{3}-\d{4}$` body = `{"name": "ada", "tags": ["a", "b"]}` print(sql) print(page) print(pattern, body) ``` ```output SELECT id, name FROM users WHERE name = $1

Hello, ada

42 bytes of SQL

^\d{3}-\d{4}$ {"name": "ada", "tags": ["a", "b"]} ``` - `"""` strings drop the line break after the opening quotes and before the closing ones, and the indentation their lines share: indent them with the code. - `` `raw` `` strings keep every character: `\d` is a backslash and a `d`, handy for `re` patterns, JSON, and Windows paths. They have no escapes, so they can't hold a backtick. ## 11. Maps `{K: V}` Keys are `int`, `str`, `bool`, or structs, enums, tuples, and arrays made of those (so not `float` or functions): `{(int, int): str}` for a grid, `{Point: bool}` for a set of points. Iteration order is insertion order. ```tern fn main(): ages = {"ada": 36, "bob": 25} ages["cy"] = 41 # insert or overwrite ages["bob"] += 1 print(ages["ada"], "bob" in ages, ages.get("zed", -1), len(ages)) ages.remove("ada") print(ages.keys(), ages.values(), ages) counts: {str: int} = {} for w in "a b a c a".split(" "): counts[w] = counts.get(w, 0) + 1 print(counts) groups: {int: [str]} = {} for w in ["hi", "yo", "hey"]: if not (len(w) in groups): groups[len(w)] = [] groups[len(w)].push(w) print(groups) ``` ```output 36 true -1 3 ["bob", "cy"] [26, 41] {"bob": 26, "cy": 41} {"a": 3, "b": 1, "c": 1} {2: ["hi", "yo"], 3: ["hey"]} ``` - Reading a missing key with `m[k]` is a runtime error; use `k in m` or `m.get(k, default)`. - `m.remove(k)` does nothing if `k` is missing. `m.keys()` and `m.values()` return arrays. - `for k, v in m:` loops over pairs; `for k in m:` loops over keys. - Write `not (k in m)` with the parentheses; there is no `not in` operator. - A map can't be changed while looping over it. Loop over a copy of its keys instead: `for k in m.keys(): m[k] = 0`. ## 12. Printing `print(a, b, ...)` prints its arguments separated by spaces, then a newline. It accepts any value. Floats show a decimal point (`2.0`) and use the shortest exact form (`0.1 + 0.2` prints `0.30000000000000004`); very large or small ones use exponent form (`1e+19`). Strings print raw at the top level and quoted inside arrays, maps, and structs. There is no `end=` or `sep=`; build a string instead. F-strings have no format specs. The `fmt` module does fixed decimals, padding, and digit grouping: ```tern import fmt fn main(): price = 1234.5 cents = 1205 print(f"total: {fmt.fixed(price, 2)}, ${fmt.cents(cents)}, {fmt.thousands(1234567)}") print(fmt.zero_pad(7, 3), "[" + fmt.pad_left("ab", 5, " ") + "]", "[" + fmt.pad_right("ab", 5, ".") + "]") ``` ```output total: 1234.50, $12.05, 1,234,567 007 [ ab] [ab...] ``` ## 13. Functions as values (closures) ```tern struct Button: label: str on_click: fn(int) str fn apply_twice(f: fn(int) int, x: int) int: return f(f(x)) fn make_adder(k: int) fn(int) int: return fn(x) x + k # a lambda: `fn(params) expression` fn main(): k = 10 add = fn(x: int) x + k # captures a copy of k now k = 99 print(add(5), apply_twice(add, 1), make_adder(3)(4)) b = Button("go", fn(n) f"clicked {n} times") print(b.on_click(2)) fn fact(n: int) int: # a nested function: a multi-line closure if n <= 1: return 1 return n * fact(n - 1) print(fact(5)) ``` ```output 15 21 7 clicked 2 times 120 ``` - A lambda is `fn(params) expression`: one expression, no `:` and no `return`. For more than one line, define a nested `fn` and pass it by name. - Parameter types may be left out when the context says them (a typed parameter, field, or variable, or `map`/`filter`/`sort_by`). Otherwise write them: `fn(x: int) x * 2`. - A closure copies the variables it uses when it is created, and can't change them. To accumulate, loop instead, or use a `mut` parameter. ## 14. Values, `mut`, and the safety rules Tern has no references, so aliasing bugs and data races are compile errors. The rules and the fix for each: | You wrote | Error says | Fix | |---|---|---| | `n += 1` where `n` is a parameter | can't modify parameter `n` | declare it `n: mut int` and call with `&x`, or return the new value | | `inc(x)` where `inc` takes `mut` | write `&x` | `inc(&x)` | | `swap(&a, &a)` or `f(&a, a)` | would see the same value twice | pass a copy: `b = a` then `f(&a, b)` | | use of `xs` after `f(xs)` where `f` takes `take` | was handed to a `take` parameter | pass `copy(xs)` | | `xs.push(...)` inside `for x in xs:` | can't modify `xs` while looping over it | `for i in 0..len(xs):`, or build a new array | | two `parallel` tasks write one variable | data race | give each task its own variable, combine after the block | | `parallel for` writes a shared total | would race | write `out[i]`, then sum `out` after the loop | Within one expression, parts run left to right, and a compound assignment evaluates its target once (`xs[f()] += 1` calls `f` once). Reading `x` and then passing `&x` to a call further along the same expression (`print(x, bump(&x))`) is a compile error: save `x` first. Copies are lazy (copy-on-write), so `b = a` is cheap; the copy happens when one of them is written. And when `a` isn't used again, its value moves instead: `ys = xs; ys.push(v)` or `xs = add(xs, v)` never copies, nor does `row = grid[i]; row[j] = 0; grid[i] = row`. `p = ps[i]` copies the element: to change the element itself, write `ps[i].x = 1` or pass `&ps[i]` to a `mut` parameter. ```tern struct Account: owner: str balance: int fn deposit(a: mut Account, amount: int): a.balance += amount fn main(): accts = [Account("ada", 10), Account("bob", 0)] first = accts[0] # a copy first.balance = 999 # changes only the copy deposit(&accts[1], 5) # changes the element in the array accts[0].balance += 1 # so does this print(accts, first.balance) ``` ```output [Account{owner: "ada", balance: 11}, Account{owner: "bob", balance: 5}] 999 ``` Copying ordinary data (numbers, strings, arrays, maps, structs, tuples, enums) makes an independent value. A few types are *handles* to something outside the value: a channel, a `task.shared`, a task, and resources such as an open file, a database, or a connection. Copying a handle gives a second name for the same thing, so a change made through either copy is seen through both: ```tern import task fn main(): a = task.shared(0) b = a # the same shared value, not a copy of it fn bump(n: mut int): n += 1 b.update(bump) print(a.get(), b.get()) ``` ```output 1 1 ``` ## 15. Parallelism ```tern fn fib(n: int) int: if n < 2: return n return fib(n - 1) + fib(n - 2) fn main(): c = 0 parallel: # each line is a task; waits for all of them a = fib(20) b = fib(21) do: # `do:` groups several lines into one task c = fib(5) c += 1 print(a, b, c) xs = [0] * 8 parallel for i in 0..len(xs): # iterations run across all cores xs[i] = i * i # iteration i may only write xs[i] total = 0 for x in xs: total += x print(xs, total) ``` ```output 6765 10946 6 [0, 1, 4, 9, 16, 25, 36, 49] 140 ``` - A variable created by a one-line task is visible after the block. Variables created inside a `do:` group are local to it; create them before the block to use them after. - The compiler rejects any task that writes a variable another task reads or writes. One exception: tasks may write different elements of an array written with literal indexes (`xs[0] = a()` in one task, `xs[1] = b()` in another), if they don't touch the array otherwise. - `parallel` needs no threads, locks, or channels. For long-running concurrent work (tasks that outlive a block, channels between them, shared state behind a lock), use the `task` module; its functions are listed at the top of `library/std/task.tn`. Tasks (`import task`) are green threads, like Go's goroutines: code in a task looks blocking, but only the task waits (`ch.recv()`, `h.join()`, `time.sleep`, socket reads), and its core runs other tasks meanwhile. Copying a `chan`, `task`, or `shared` value shares it. Values sent through a channel are copied (cheaply: copy-on-write). ```tern import task fn fetch(id: int) str: task.sleep(10) # parks this task only; others keep running return f"page {id}" fn produce(ch: chan[int], n: int): for i in 1..n + 1: ch.send(i * i) ch.close() # ends the receiver's `for` loop fn main(): hs: [task[str]] = [] for id in 0..3: hs.push(task.spawn(fn() fetch(id))) # all three run at once for h in hs: print(h.join()) # waits; an error in the task is raised here ch: chan[int] = task.channel(8) # holds up to 8 values (0: unbounded) task.spawn(fn() produce(ch, 5)) total = 0 for x in ch: # until closed and empty total += x print(total) hits = task.shared(0) # shared[int]: many tasks, one value fn bump(n: mut int): n += 1 ws: [task[void]] = [] for i in 0..100: ws.push(task.spawn(fn() hits.update(bump))) # one update at a time for w in ws: w.join() print(hits.get()) ``` ```output page 0 page 1 page 2 55 100 ``` - `task.spawn` takes a function with no parameters; wrap a call in a lambda to pass arguments. The task gets copies of what the lambda uses, as with any closure. - A channel needs its type written: `ch: chan[int] = task.channel(8)`. `ch.recv()` on a closed, empty channel is a runtime error; `for x in ch:` and `ch.next(&x)` stop instead. - A runtime error in a task ends that task; `h.join()` raises it again. Other tasks keep running. The program ends when `main` returns; if a task failed and nobody joined it, its error is printed and the program exits with status 2 (tasks spawned inside web handlers don't count). `h.cancel()` on a task that already failed discards its error. - `task.local_set(key, value)`, `task.local_get(key) str` (`""` if unset) and `task.local_clear()` keep strings that belong to the running task (a request id, tracing context). A spawned task starts with a copy of its spawner's. `task.id() int` names the running task (0 outside tasks). - `os.on_error(f)` calls `f(os.ErrorReport)` for every runtime error from then on, caught ones too (a failed task, a web handler's `500`): for error trackers. See [os](std/os.md). - Code inside `parallel:` may use channels too; there it waits like a thread. ### Groups of tasks: `task.all`, `task.try_all` (Go's errgroup) `task.all(fs)` runs every function in `fs` (`[fn() T]`) at once and returns their results in order; a runtime error in one is raised in the caller. `task.try_all(fs)` does the same for functions that return `Result[T, E]`: `Ok` with every value in order, or the first `Err` to arrive, after cancelling the tasks still running. `task.try_all_limit(fs, n)` runs at most `n` at a time (Go's `SetLimit`), for "fetch these 500 URLs, 8 at once". ```tern import task fn fetch(id: int) Result[str, str]: if id == 13: return Err(f"page {id}: not found") return Ok(f"page {id}") fn main(): print(task.all([fn() 2 + 2, fn() 3 * 3])) good: [fn() Result[str, str]] = [] for id in 0..4: good.push(fn() fetch(id)) print(task.try_all_limit(good, 2)) bad: [fn() Result[str, str]] = [fn() fetch(1), fn() fetch(13)] print(task.try_all(bad)) # the first Err; any still running are cancelled ``` ```output [4, 9] Ok(["page 0", "page 1", "page 2", "page 3"]) Err("page 13: not found") ``` ### Counters and caches: `task.atomic`, `task.shared_map` `shared[T]` changes any value safely, but each `get` and `update` takes a lock that tasks wait for in turn: fine for occasional changes, slow for a counter or cache that every request touches. For those two cases there are faster handles (copies share one, like channels): - `task.atomic(v)`: an `int` changed with atomic instructions. `a.load()`, `a.store(v)`, `a.add(n)` (returns the new value), `a.cas(old, new)` (stores `new` if the value was `old`). - `task.shared_map()`: a `str -> str` map any task can use at once (`m.put(k, v)`, `m.get_or(k, default)`, `m.has(k)`, `m.remove(k)`, `m.count()`), plus an `int` per key for counting (`m.add_int(k, n)`, `m.get_int(k)`). Under contention they are hundreds of times faster than `shared[T]`. Copies share the same one (like a channel), and the last copy gone frees it, entries and all: a cache can start again with a fresh map. ```tern import task fn visit(hits: task.Atomic, pages: task.SharedMap, path: str) int: pages.add_int(path, 1) return hits.add(1) fn main(): hits = task.atomic(0) pages = task.shared_map() pages.put("/", "home") hs: [task[int]] = [] for i in 0..100: p = "/" if i % 2 == 0 else "/about" hs.push(task.spawn(fn() visit(hits, pages, p))) for h in hs: h.join() print(hits.load(), pages.get_int("/"), pages.get_int("/about"), pages.get_or("/", "?")) ``` ```output 100 50 50 home ``` ## 16. Standard library Import a module with `import name`, then call `name.fn(...)`. Or `from name import fn1, Type`. | Module | Functions and types | |---|---| | `fmt` | `fixed(x: float, digits: int) str` (`"3.14"`), `cents(n: int) str` (`1205` → `"12.05"`), `zero_pad(n, width) str`, `pad_left(s, width, fill) str`, `pad_right(s, width, fill) str`, `thousands(n) str` (`"1,234,567"`) | | `math` | `sqrt floor ceil round log exp sin cos tan (x: float) float`, `pow(x: float, y: float) float`, `pi() float` | | `random` | `int(lo: int, hi: int) int` (inclusive), `float() float` in `[0, 1)` | | `time` | `now() float` (seconds since 1970), `now_ms() int`, `sleep(ms)`, `mono_ns() int`, `mono_ms() int`; `rfc3339(t) str`, `rfc3339_local(t) str`, `parse_rfc3339(s) float`; `format(t, layout) str` / `format_local` (strftime + `%L` ms, `%f` us); `parse(s, layout) float` / `parse_local`; `utc(t) Date`, `local(t) Date` (year month day hour minute second nano weekday yday offset); `date(y, mo, d, h, mi, s) float` (UTC; out-of-range values carry); `offset(t) int`; `duration(secs) str` ("1h2m3.5s"), `parse_duration(s) float`; `error() str` (a failed parse returns 0.0); Result forms: `try_parse_rfc3339(s)`, `try_parse(s, layout)`, `try_parse_duration(s)` (`Result[float, str]`) | | `os` | `args() [str]` (`args()[0]` is the program), `env(name: str, default: str) str`, `exit(code: int)`, `eprint(s)`, `read_line() str`, `read_all() str`, `run(cmd, args) Output` / `run_input(cmd, args, input)` (status out err), `spawn(cmd, args) Process` / `spawn_with(cmd, args, ProcOpts{dir env out err})`: `p.ok() p.write(s) p.close_input() p.out_line() p.out_all() p.wait() p.wait_ms(ms) p.kill(sig) p.pid()`; `cwd() chdir(dir) setenv(k, v) error()`; `wait_signal() str` (waits, task-friendly, for SIGINT/SIGTERM/SIGHUP/SIGUSR1/SIGUSR2), `flush()`, `cpu_count() int`, `load_avg() float` (-1.0 where unavailable), `getpid() int`, `executable() str` (this program's absolute path); `on_error(f: fn(ErrorReport))`, `last_error_report() ErrorReport` (message, file, line, function, frames, build id, task: for error trackers), `at_exit(f: fn())` (f runs when the program ends, the last added first) | | `fs` | `read(path) str` (`""` if unreadable), `write(path, text) bool`, `append(path, text) bool`, `exists(path) bool`, `remove(path) bool`, `list(dir) [str]`, `list_marked(dir) [str]` (directories end in `/`), `mkdir(path) bool`, `chmod(path, mode) bool` (`0o600`), `rename(a, b) bool`, `stat(path) Info` (exists is_dir size modified), `walk(dir) [str]`, `copy(a, b) bool`, `remove_all(path) bool`, `temp_dir(prefix) str`, `error() str`; files: `open(path) File` / `create` / `open_append`, `f.ok()`, `f.read_line() str`, `f.read_some(max) str`, `f.put(s) bool`, `f.flush() bool`, `f.close() bool`; Result forms: `try_read(path) Result[str, str]`, `try_write`, `try_append`, `try_list`, `try_stat`, `try_open` | | `log` | leveled structured logs on stderr: `debug/info/warn/error(msg)`, `debug_with/info_with/warn_with/error_with(msg, fields: {str: str})`; `set_level(name) bool` ("debug" "info" "warn" "error" "off"), `set_format(name) bool` ("logfmt" "json" "text"); env `TERN_LOG_LEVEL`, `TERN_LOG_FORMAT`; `recent(n) [Record]`, `keep(level)` | | `json` | `encode(any value) str`, `decode(text: str, out: mut T) bool` (all-or-nothing: on `false`, `out` is unchanged); an `Option[T]` field is `null` or the value (missing: `None`); an enum is `"Name"` or `{"Name": payload}` (names decode loosely: `"in_progress"` is `InProgress`), `schema[T]() str` (T's JSON Schema, made while compiling); `parse(text) Result[Value, str]` for JSON of unknown shape: `v.field(k)`, `v.at(i)` (Null when missing, so they chain), `v.str_or(d)`, `int_or`, `float_or`, `bool_or`, `v.fields()`, `v.elems()`, `v.size()`, `v.to_map()` (`{str: Value}`, for many lookups in a big object), `dump(v) str`; `Value` is `Null`, `Bool(b)`, `Num(x, text)`, `Str(s)`, `Arr(items)`, `Obj(keys, vals)` | | `sqlite` | SQLite (package). `open(path) sqlite.Db`; `d.exec(sql, params: [str]) int` (last insert id); `d.exec_count(sql, params) int` (rows changed); `d.query(sql, params) [{str: str}]`; `d.query_into(sql, params, &structs_or_struct) int` (rows read, through std `sql`); `d.exec_later(sql, params)` + `d.flush()` (writes committed in the background); `sqlite.NULL` (a NULL parameter); `d.run_script(sql) Result[int, str]` (several statements; errors returned); `d.describe(sql) Result[Statement, str]` (params, columns: name decl_type table column not_null); `query_into` fields may be `Option[T]` (NULL is None) | | `task` | `spawn(f: fn() T) task[T]`; `h.join() T`, `h.done() bool`, `h.cancel()`; `channel(n) chan[T]` (give the variable its type; `n = 0`: unbounded); `ch.send(v)`, `ch.recv() T`, `for x in ch:`, `ch.next(&x) bool`, `ch.try_recv(&x) bool`, `ch.recv_timeout(&x, ms) bool`, `ch.close()`, `len(ch)`; `select(chans: [chan[T]], &x, ms) int` (index of the channel x came from; `-1` timed out, `-2` all closed); `shared(v) shared[T]`, `s.get() T`, `s.set(v)`, `s.update(f: fn(mut T) R) R`; `sleep(ms)`, `yield_now()`; `blocking(f: fn() T) T` (runs f on a pool thread: for blocking work like fsync or SQLite); `local_set(key, value)`, `local_get(key) str`, `local_clear()` (the running task's strings; spawned tasks inherit a copy); `id() int`; `thread_slot() int` (the running thread's scheduler slot: for per-thread caches); `workers() int` (the scheduler's threads: how many `parallel` iterations run at once) | | `net` | TCP for tasks. `listen(port) Listener`, `listen_on(host, port) Listener`, `l.accept() Conn`, `l.port() int`, `l.stop()`; `connect(host, port) Conn` (`c.ok()`, else `net.error() str`); `c.read(max) str`, `c.read_line() str` (with its `"\n"`; `""` at the end), `c.read_exact(n) str`, `c.write(s) bool`, `c.close()`, `c.peer() str`; `c.pending() bool` (a read wouldn't wait), `c.refill(&buf, used, max) int` (drop `buf[..used]`, read up to `max` bytes onto the end: a protocol parser's buffer without copies), `c.starttls(host, verify) bool` (needs `import tls`); `connect` resolves names off the worker (cached, `TERN_DNS_TTL`=30s, at most 16k names; `TERN_DNS_SERVERS=1.1.1.1,8.8.8.8` asks those servers over UDP from the task itself, within the request's deadline, keeping localhost, `.local` and /etc/hosts names for the system); `dial(host, port) Result[Conn, str]` | | `roost` | axum-style framework on tasks. `router() Router`; `r.get/post/put/delete/patch(path, h)`, `r.route(method, path, h)`, `r.any`, `r.nest(prefix, sub)`, `r.fallback(h)`, `r.ws(path, fn(req, ws))`, `r.sse(path, fn(req, out))`, `r.layer(mw)`, `r.wrap(mw)`; `serve(r, port)`, `start(r, port) Server`; handlers `fn(roost.Request) roost.Response`; `req.param/param_int/query/header/cookie/form/json(&v)`; `text html json status not_found redirect`; middleware `logger cors request_id timeout bearer_auth auth gzip`; client `get(url) post(url, body, type) fetch(ClientRequest)`; `serve_tls(r, port, cert, key)`; `req.peer()`, `req.client_ip()`, `roost.trust_proxies(n)`, `rate_limit(per_second, burst)`, `rate_limit_by(per_second, burst, key)`, `r.files(prefix, dir)`, `file_response(path, req)`, `req.parts()` / `req.part(name)`, `r.post_stream` / `r.put_stream` with `req.read_body(max)`, `save_body(path)`, `next_part(&p)`, `read_part(max)`, `save_part(path)`, `r.max_connections(n)`; `r.metrics(path)` (Prometheus), `metric_add/metric_set(name, labels, v)`; validation `check()` with `required/min_len/max_len/between/one_of/email/fail/valid/problems`; SIGINT/SIGTERM make `serve` drain in-flight requests (`TERN_WEB_DRAIN_SEC`=10) and return. See `library/roost/roost.tn`; `try_get(url)`, `try_fetch(req)` (`Result[Response, str]`); `req.matched_route()` (the route pattern, once `roost.track_routes()` was called) | | `crypto` | `sha256(s) str`, `sha512(s) str` (raw bytes), `hmac_sha256(key, msg) str`, `hmac_sha512`, `sha1`, `hmac_sha1` (for TOTP); `hash_password(pw) str` (Argon2id PHC string), `verify_password(pw, hash) bool`; `argon2id(pw, salt, t, m_kib, p, len) str`, `pbkdf2_sha256(pw, salt, iters, len) str`; `random_bytes(n) str`, `random_int(lo, hi) int` (OS secure random); `equal(a, b) bool` (constant time); `hex(s)`, `unhex(s)`, `base64(s)`, `base64_decode(s)`, `base64url(s)`, `base64url_decode(s)`; `uuid4() str`, `uuid7() str`; `error() str` (why a decode failed); Result forms: `try_unhex(s)`, `try_base64_decode(s)`, `try_base64url_decode(s)`; `seal(key, plaintext) str` / `unseal(key, sealed) str` / `try_unseal` (ChaCha20-Poly1305, 32-byte key), `chacha20_poly1305(key, nonce, aad, pt)` / `chacha20_poly1305_open(...)` | | `compress` | zstd and libdeflate, built in (linked only when imported). `zstd(data, level) str` (one frame; levels 1..19, 3 is the default, up to 22 or negative for faster; clamped), `unzstd(data) Result[str, str]` (every frame in data, with or without its size in the header), `unzstd_max(data, max_out) Result[str, str]` (Err past max_out bytes: for untrusted input), `zstd_bound(n) int`, `error() str`. Any bytes (NULs included); up to 4GB each way; `gzip(data, level) str` (levels 1..12, 6 default, 0 stores), `gunzip(data)` (every member) / `gunzip_max(data, max_out)` (`Result[str, str]`), `zlib`/`unzlib`/`unzlib_max`, `deflate`/`inflate`/`inflate_max` (raw) | | `re` | regular expressions, linear time (RE2 syntax: no backreferences or lookaround). `compile(pattern) Result[Regex, str]`, `must(pattern) Regex`; `r.test(s) bool`, `r.full(s) bool`, `r.search(s) Option[Match]`, `r.search_at(s, at)`, `r.find_all(s) [Match]`, `r.count(s) int`, `r.replace_all(s, repl) str` (`$1` `${name}` `$0` `$$`), `r.replace_first(s, repl)`, `r.replace_with(s, fn(Match) str)`, `r.split_all(s) [str]`, `r.names() [str]`; `escape(s) str`, `error() str`; `literals(pattern, min) Result[[[str]], str]` (literals every match contains, for searching an index first). `Match`: `start end text groups starts named`, `m.group(name) str`. Patterns are strings: `"\\d+"` | | `simd` | 16 bytes at once (NEON, SSE, wasm SIMD): `load(s, i) bytes16`, `load_upto(s, i)` (zeros past the end), `of(t)`, `splat(b)`, `to_str(v)`; `& | ^ ~ + - << >> ==` lane by lane; `eq(a, b)`, `lt`, `le`, `min`, `max`, `lookup(t, i)` (a 16-entry table per lane), `bits(v) int`, `first_lane(v) int`, `any(v) bool`, `lane(v, k) int`, `first(m) int`. See [simd](std/simd.md) | | `flag` | command-line flags into a struct (a field per flag: `min_len` is `--min-len`; the struct's values are the defaults): `cli(name, about) Cli`, `c.flag(field, short, help)`, `c.env(field, var)`, `c.require(field)`, `c.args(text)`, `c.stop_at_args()`, `c.exit_on_help`; `parse(c, &out) Result[[str], str]` (os.args(); returns the positionals), `parse_from(c, args, &out)`, `usage(c, out) str`. Fields: int, float, str, bool (`--x`, `--no-x`, `-abc`), an enum (any case), `[T]` (repeatable), `Option[T]`; `--help` prints the usage and exits 0 | | `term` | `is_tty()`, `stderr_is_tty()`, `stdin_is_tty()`, `width()`, `height()`; `color() bool`, `set_color(on)`; `red green yellow blue magenta cyan gray bold dim italic underline (s) str` (plain text unless stdout is a terminal, `NO_COLOR` unset; `FORCE_COLOR=1`); `strip_ansi(s)`, `display_width(s)`, `pad(s, width)`, `table(rows: [[str]]) str`, `progress(done, total, label)` (stderr, only on a terminal), `clear_line()`, `secret(prompt) str` (no echo), `read_line(prompt) str` (line editing and history on a terminal) | | `heap` | min or max heap, `T` int/float/str: `min_heap[T]()`, `max_heap[T]()`, `heap_of(xs, max) Heap[T]` (O(n)), `h.push(x)`, `h.pop() Option[T]`, `h.peek() Option[T]`, `h.len()`, `h.is_empty()`, `h.drain() [T]` | | `pqueue` | priority queue of any `T` by a priority `P` (int/float/str): `queue[P, T]()` (lowest first), `queue_max[P, T]()`, `q.push(priority, value)`, `q.pop() Option[T]`, `q.peek()`, `q.peek_priority() Option[P]`, `q.len()`, `q.is_empty()` | | `deque` | ring-buffer deque: `new[T]()`, `of(xs)`, `d.push_back(x)`, `d.push_front(x)`, `d.pop_back()` / `d.pop_front()` (`Option[T]`), `d.front()`, `d.back()`, `d.at(i) T`, `d.put(i, x)`, `d.len()`, `d.is_empty()`, `d.to_array()`, `d.clear()` | | `bitset` | `new(n) Bitset` (grows), `b.set(i)`, `b.unset(i)`, `b.toggle(i)`, `b.has(i) bool`, `b.count()` (popcount), `b.ones() [int]`, `b.size()`, `b.union_with(o)`, `b.intersect_with(o)`, `b.subtract(o)` | | `url` | URLs by Go's net/url rules: `parse(text) Result[Url, str]`, parts as methods (`u.scheme() user() password() host() port() path() raw_query() fragment() opaque()`, plus `escaped_path() escaped_fragment() raw_path() raw_fragment() force_query() omit_host()`), setters (`u.set_path(p)`, `set_host`, `set_port`, `set_raw_query`, `set_fragment`, `set_scheme`, `set_user`, `set_password`), `u.parts() Parts` / `build(p: Parts) Url` for several at once, `format(u) str`, `resolve(base, ref: Url) Url`, `join(base, ref: str) Result[str, str]`, `u.host_port()`, `u.request_uri()`, `u.query() Query`; `parse_query(raw) Result[Query, str]`, `new_query()`, `q.get(k) str`, `q.get_all(k) [str]`, `q.has(k)`, `q.add(k, v)`, `q.set(k, v)`, `q.remove(k)`, `encode_query(q) str`, `query_string(pairs: {str: str}) str`; `query_escape`/`query_unescape`, `path_escape`/`path_unescape` (unescapes return `Result`) | | `binary` | Binary formats, bytes in a `str`: `writer() Writer` with `w.put_u8/u16be/u16le/u32be/u32le/u64be/u64le(v)`, `put_f32be/f32le/f64be/f64le(v)`, `put_uvarint(v)`, `put_varint(v)` (zigzag), `put_bytes(s)`, `len()`, `bytes() str`, `patch_u32be(i, v)`; `reader(data) Reader` with `r.read_u8()` ... `read_u64le()`, `read_i8/i16be/i16le/i32be/i32le()`, `read_f32be()` ... `read_f64le()`, `read_uvarint()`, `read_varint()`, `read_bytes(n)`, `skip(n)`, `has(n) bool`, `left() int`, `ok() bool`, `error() str` (a read that doesn't fit gives 0 and every later one too; the first problem is kept); at an offset: `u8(s, i)` ... `u64le(s, i)`, `i16be` ... `i32le`, `f32be` ... `f64le`, `set_u8(&s, i, v)` ... `set_f64le` (past the end: a runtime error); `uvarint_len(v)`, `varint_len(v)` | | `toml` | TOML 1.0: `decode(text, &out) Result[int, str]` (fills a struct by field name, like `json.decode`), `parse(text) Result[json.Value, str]`, `encode(value) Result[str, str]` (a struct or map), `dump(v: json.Value) Result[str, str]`; dates and times are RFC 3339 strings; errors name the line | | `path` | paths as text, `/`-separated (Go's `path` rules): `join(a, b) str`, `join_all(parts) str`, `resolve(base, p) str` (an absolute p wins), `clean(p) str`, `dir(p) str`, `base(p) str`, `split(p) (str, str)`, `ext(p) str` (`".gz"`; none for `.bashrc`), `stem(p) str`, `with_ext(p, ext) str`, `parts(p) [str]`, `is_abs(p) bool`, `abs(p) str`, `rel(base, target) Result[str, str]`, `matches(pattern, name) bool` (`* ? [a-z] [!a-z] \x`), `glob(pattern) [str]` (sorted; `**` is any number of directories; hidden names only for a piece starting with `.`) | | `csv` | RFC 4180 CSV, like Go's `encoding/csv`. `parse(text)` (`Result[[[str]], str]`), `records(text)` (`Result[[{str: str}], str]`, the first row is the header), `reader(text) Reader` + `r.next(&row) bool` / `r.err` / `r.line` (one record at a time), `format(rows) str`, `format_row(row) str`; `options() Options` (`sep comment fields lazy_quotes trim_leading_space crlf`), `tsv()`, and `parse_with` / `records_with` / `reader_with` / `format_with` / `format_row_with` taking them; errors name the line and column | | `sql` | What database drivers share (Go's `database/sql`): `decode_rows(raw, &out) int` fills a struct or `[struct]` from a driver's rows (columns to fields by name; int, float, bool, str, label enums, arrays, JSON fields, Options; `-1` and `decode_error() str` when a value doesn't fit), `rows(columns) Rows` with `r.row(n)`, `put_int/put_float/put_bool/put_text/put_null`, `bytes()` builds them, `rows_from(head)` starts one from an earlier header; `row_field(&out, name)`, `row_id(&out)` and `set_int/set_float/set_text/set_null(&out, field, v)` fill the first row of a struct from values in hand; `TEXT INT FLOAT BOOL UINT TAGGED` name the formats ([std/sql.md](std/sql.md)) | | `html` | escaping and templates that escape by default. `escape(s) str`; `render(template, data_json) str`, `render_file(path, data_json) str` (parsed once, reloaded on change; `{{> file}}` includes); `error() str`; Result forms `try_render(template, data_json)`, `try_render_file(path, data_json)` (`Result[str, str]`). Tags: `{{x}}` `{{a.b}}` `{{.}}` escaped, `{{{x}}}` raw, `{{json x}}`, `{{#each xs}}..{{/each}}` with `{{@index}}`, `{{#if x}}..{{else}}..{{/if}}`, `{{#unless x}}`, `{{! comment}}`; data is `json.encode(value)` | | `tls` | TLS (aws-lc, which `tern` builds once per target and links only when imported; `TERN_TLS=system` uses the system's OpenSSL); importing it also enables `https://` in `http.get/fetch` and `s.listen_tls(...)` / `roost.serve_tls(r, port, cert, key)` / `roost.start_tls(r, host, port, cert, key) Server`. `connect(host, port) net.Conn` (verified; `c.ok()`, else `tls.error()`), `connect_insecure(host, port)`, `upgrade(c, host) bool` (STARTTLS), `server(cert, key) Server`, `accept(c, s) bool`, `version(c) str`, `trust(ca_file) bool`; `dial(host, port) Result[net.Conn, str]`; ALPN: `set_alpn(s, protocols: [str])` (server, preferred first), `connect_alpn(host, port, protocols) net.Conn`, `alpn(c) str` (the protocol agreed, e.g. `"h2"`) | | `testing` | `eq_int eq_float eq_str eq_bool (got, want)`, `eq_ints eq_strs`, `near(got, want, tol)`, `is_true(cond, what)`, `is_false`, `includes(s, sub)`, `fail(msg)`: checks for `tern test` ([cli.md](cli.md#testing)) | | `queue` | client for the `ternq` broker (`examples/queue/`): `connect(host, port) Client`, `connect_many(["h1:7070,h2:7071"])` (shards; each with failover addresses), `connect_auth(addrs, token)`, `c.publish(q, body) bool`, `c.publish_batch`, `c.subscribe(q, group, prefetch) Sub`, `s.next(&m) bool`, `c.ack(m)`, `c.nack(m)`, `c.fetch(s, max)`, `c.fetch_wait(s, max, ms)`, `c.stats(q)`. See `library/queue/queue.tn` | | `openapi` | OpenAPI 3.1 from web routes (package). `new(title, version, description) Spec`, `bearer(&s, desc)`, `server(&s, url)`; `op(method, path, summary) Op` then `o.returns[T]()`, `o.accepts[T]()`, `o.returns_text()`, `o.status(n)`, `o.query(name, type, desc)`, `o.describe(t)`, `o.tag(t)`, `o.id(t)`, `o.fails(code, desc)`, `o.public()`; `add(&s, &r, o, handler)` registers the route too, `record(&s, o)` only documents; `document(s) str`, `typescript(s) str` (types + `createClient`), `serve(s, &r)` (`/openapi.json`, `/docs`) | | `pg` | PostgreSQL. `pool(url, size) Pool`, `connect(url) Pool`; `p.ok()`, `pg.error() str`; `p.exec(sql, params: [str]) int` (rows; -1 on error); `p.query(sql, params) [{str: str}]`; `p.query_json(sql, params) str` / `p.query_row_json(...)` (typed JSON for `json.decode(…, &structs)`); `p.begin()`, `p.commit()`, `p.rollback()` (transactions bound to the task); `p.stats() PoolStats` (size open idle in_use waiting waits wait_ms timeouts opened closed), `p.prune()`, `p.close()`; URL `&pool_timeout=30s&max_lifetime=1h&max_idle=30m&statement_cache=off&application_name=x`. Params are `$1, $2`; `?sslmode=require` needs `import tls`; `fetch[T](sql, params) Result[[T], str]`, `fetch_one[T](sql, params) Result[Option[T], str]`, `pg.NULL` (a NULL parameter), `array_int(xs)` / `array_float` / `array_bool` / `array_str` (a list as one parameter: `= ANY($1)`), rows fill `[int]` `[str]` ... fields from PostgreSQL arrays, `run(sql, params) Result[int, str]`, `run_script(sql) Result[int, str]` (several statements, simple protocol), `describe(sql) Result[Statement, str]` (param type names; columns: name type_name type_oid kind base_type table not_null); `Option` fields decode NULL as None | | `redis` | Redis client (package). `connect(url) Client` / `pool(url, size)`; `c.ok()`, `redis.error()`; `c.get_str(k) str`, `c.set_str(k, v) bool`, `c.set_ex(k, v, secs)`, `c.set_nx(k, v)`, `c.del(k) int`, `c.exists(k) bool`, `c.incr(k) int`, `c.incr_by(k, n)`, `c.expire(k, secs)`, `c.ttl(k)`, `c.mget(keys) [str]`, `c.publish(ch, msg) int`; `c.cmd(args: [str]) Reply` (kind text num items), `c.pipeline([[str]]) [Reply]`; `c.subscribe(channels) Subscription`, `sub.next(&m) bool`, `sub.unsubscribe()`; `c.close()`; `c.try_get(k) Result[Option[str], str]`, `c.try_cmd(args) Result[Reply, str]` | | `auth` | JWT HS256 and signed-cookie sessions (package). `jwt_for(subject, secret, ttl) str`, `jwt_encode(payload_json, secret) str`, `jwt_verify(token, secret) Token` (ok payload subject error), `require_jwt(secret)` middleware (ctx "jwt.sub"); `session(req, secret) {str: str}`, `set_session(&res, secret, data)`, `set_session_for(&res, secret, data, max_age, attrs)`, `clear_session(&res)`; `verify(token, secret) Result[Token, str]` | | `migrate` | schema migrations (package) for PostgreSQL and SQLite from `VERSION_NAME.sql` files (`-- migrate:up`, `-- migrate:down`, `-- migrate:no-transaction`). `postgres(p) Target`, `sqlite(d) Target`, `open(url) Result[Target, str]`; `t.up(dir)`, `t.down(dir, n)` (`Result[[Migration], str]`: what ran), `t.status(dir) Result[[Status], str]` (state: applied pending changed missing); `load(dir)`, `parse(version, name, text)`, `t.apply(ms)`, `t.rollback(ms, n)`, `t.report(ms)`; `create(dir, name) Result[str, str]`; `cli(dir, url) Result[int, str]` (up, down [N], status, new NAME); `label(m)`, `label_of(s)`, `split_statements(sql) [str]`. Each migration runs in a transaction with its bookkeeping; runs are serialized (pg advisory lock); an edited applied migration is an error | | `sqlgen` | typed functions from SQL (package), like sqlc: `-- name: get_user :one` (or `:many`, `:exec`) above each query, `@name` parameters (`@x:int` to give a type, `@x?` for an `Option` that may be NULL), `"col!"` / `"col?"` / `"n: int"` column notes. `cli(url) Result[int, str]` (`QUERIES.sql ... -o OUT.tn [--migrations DIR] [--check]`), `generate(dbase, files) Result[str, str]` with `postgres(p)` / `sqlite(d)`; writes a Row struct per query and `fn q(pool: pg.Pool, ...) Result[Option[Row] | [Row] | int, str]` (SQLite: `fn q(conn: sqlite.Db, ...) Option[Row] | [Row] | int`); nullable columns are `Option` | | `grpc` | gRPC over HTTP/2, over TLS (ALPN `h2`) or cleartext (h2c) (package): unary calls, HPACK with Huffman, many streams per connection. `start(host, port, handler: fn(Call) Reply, options()) Server`, `start_tls(host, port, handler, options(), cert, key) Server`, `serve(port, handler)`, `port(s)`, `wait(s)`, `stop(s, drain_ms)` (GOAWAY, then drains); `Call` has `method` (`"/pkg.Service/Method"`), `metadata` `{str: str}`, `message` (the request's bytes), `encoding` (`"gzip"` when compressed; accepted ones in `Options.encodings`); reply with `ok(body)` or `fail(status, msg)` (`grpc.OK` ... `grpc.UNAUTHENTICATED`); `Options`: `max_message max_frame max_header_list max_streams stream_window conn_window header_table encodings`; client `call(host, port, method, metadata, message) Reply` (h2c), `call_tls(...)` (TLS, verified). Imports `tls`. No streaming RPCs | | `sentry` | error reporting to Sentry (package). `options() Options` (release environment server_name sample_rate max_queue breadcrumbs timeout_ms scrub), `init(dsn, o) Client`; `c.ok()`, `c.error()`; `c.capture_message(msg, level) str`, `c.capture_error(r: os.ErrorReport) str`, `c.capture_result(r: Result[T, str]) str` (the event id); `install(c)` (every runtime error, via `os.on_error`), `middleware(c)` (request data and route); `set_tag(k, v)`, `set_extra(k, v)`, `set_user(id, email)`, `clear_scope()` (task-local); `c.flush(ms) bool`, `c.close()`, `c.sent()`, `c.dropped()` | Errors in the Result forms read `: : ` (`fs: read /tmp/x: no such file or directory`). | `http` | `serve(port, handler)` / `serve(port, handler, app)` / `serve(port, handler, &app)`; `Request{method, path, raw_query, head, body, params, ctx, conn}`; `Response{status, content_type, body, headers, upgrade}`; `text(s) html(s) json(s) status(code, s) not_found() no_content() redirect(url) with_status(res, code)`; `req.query(name) req.queries() req.header(name) req.headers() req.cookie(name) req.form(name) req.parts() req.param(name) req.json(&v)`; `res.set_header(name, v) res.set_cookie(name, v) res.get_header(name)`; `new_request(method, target, body) Request`, `req.add_header(name, v)` (tests); `match(path, "/users/{id}", ¶ms) bool`; engine: `new_server() Server`, `s.add_route(method, pattern, h, ws, sse, stream)`, `s.listen(host, port, fallback)` / `s.listen_tls(..., cert, key)`, `s.wait()`, `s.stop()`, `ws_upgrade`, `sse_open`, `static_file(dir, rel, req)`; client: `get(url) post(url, body, type) fetch(request(method, url))`, `fetch_max(cr, max_body)` (fails past max_body bytes; `Connection: close` in cr.headers keeps it out of the pool); idle pooled connections close after `TERN_HTTP_POOL_IDLE`=30 s, at most `TERN_HTTP_POOL_MAX`=512 wait in all, `try_get`, `try_fetch`, `failed(res)`; middleware helpers: `client_ip limiter request_id gzip metrics_text` | Builtins (no import): `print len push pop copy abs min max sqrt float int str chr` and the string, map, and array functions in §8 to §11. Two more for web code: `json_quote(s) str` (`s` as a JSON string literal, quotes and escapes included: `json_quote("a\"b")` is `"a\"b"` with the quotes) and `url_decode(s) str` (`%20` and `+` become spaces). `min`/`max` take two numbers of the same type. `abs` takes an int or a float; `sqrt` takes a float (`sqrt(float(n))`). ```tern import math import json import fs struct Config: name: str workers: int tags: [str] fn main(): print(math.sqrt(16.0), math.floor(2.7), math.pow(2.0, 10.0), min(3, 4), max(1.5, 0.5)) c = Config("svc", 4, ["a"]) text = json.encode(c) print(text) back = Config("", 0, []) if json.decode(text, &back): print(back.workers) bad = Config("keep", 1, []) print(json.decode("{not json", &bad), bad.name) path = "/tmp/tern_doc_test.txt" fs.write(path, "line1\nline2\n") lines = fs.read(path).trim().split("\n") print(lines, fs.exists(path)) fs.remove(path) ``` ```output 4.0 2.0 1024.0 3 1.5 {"name":"svc","workers":4,"tags":["a"]} 4 false keep ["line1", "line2"] true ``` A web server with a database (build with `tern build app.tn -o app`): ```text import http import sqlite import json struct Todo: id: int title: str fn handle(req: http.Request, d: sqlite.Db) http.Response: params: {str: str} = {} if req.method == "GET" and req.path == "/todos": todos: [Todo] = [] d.query_into("SELECT id, title FROM todos", [], &todos) return http.json(json.encode(todos)) if req.method == "POST" and http.match(req.path, "/todos/{title}", ¶ms): id = d.exec("INSERT INTO todos (title) VALUES (?)", [params["title"]]) return http.json(json.encode(Todo(id, params["title"]))) return http.not_found() fn main(): d = sqlite.open("todos.db") d.exec("CREATE TABLE IF NOT EXISTS todos (id INTEGER PRIMARY KEY, title TEXT)", []) http.serve(8080, handle, d) ``` - The handler is `fn(req: http.Request) http.Response`, or takes a second parameter `app` (shared read-only by all threads), or `app: mut App` with `http.serve(port, handle, &app)` (requests that change it run one at a time). - A runtime error inside a handler becomes a 500 response; the server keeps running. - SQL parameters are always strings (`[str(id)]`) and are bound safely. - Each server thread opens its own connection, so `sqlite.open(":memory:")` gives every thread a separate, empty database. Servers need a file path. - A `sqlite.Db` is a handle, so a read-only `d: sqlite.Db` parameter can still run `INSERT`s: the database is outside the program's values. "Read-only" is about Tern values only. ## 17. Modules and packages Every `.tn` file is a module. `import models` looks in the standard library and the first-party packages (`roost`, `queue`, `pg`), then for `models.tn` or `models/` (a directory of `.tn` files forming one package) next to the main file, then in `deps/`. Use `models.User`, `models.load(...)`, or `from models import User, load`. Names starting with `_` are private. Third-party packages are git repositories: list them in a `tern.mod` file next to your program and run `tern get`, which vendors them into `deps/`, where `import` finds them (see [cli.md](cli.md#packages)). ## 18. Other targets and C - `--target wasm | ios | ios-sim | android`; `build --lib` makes a C library with a header from `export fn` functions; `--shared` makes a `.so`/`.dylib`. On wasm, `--lib` also writes a JavaScript loader. Wasm runs `parallel` sequentially; `sqlite` and `http` aren't available there. - Calling C: an `extern "header.h":` block of `fn` signatures (see README.md, "Calling C"). Only `int float bool str ptr [int] [float]` and functions of those cross the boundary. `ptr` is an opaque C pointer; `null` is the null pointer (`p: ptr = null`) and `p.is_null()` tests for it. `null` is only a `ptr`: other types have no null. A `str` result stops at the first NUL byte; for binary data, return a `ptr` and a length and copy it with `string_at(p, n) str` (see guides/c-interop.md). ## 19. What Tern does not have (and what to write instead) | From another language | In Tern | |---|---| | `def f(x):`, `func`, `function`, `fun` | `fn f(x: int) int:` | | `class`, methods, `self`, `impl`, interfaces, traits | `struct` + functions taking it first; `x.f()` calls `f(x)` | | `let x`, `var x`, `const X`, `x := 1`, `auto` | `x = 1` | | global variables | `const MAX_USERS = 100` for fixed values; `const CACHE = task.shared_map()` for a program-wide cache; pass a struct for other state that changes | | `None`, `null`, `nil`, `Option`, `T?` | `Option[T]`: `Some(x)` or `None` (§9) | | exceptions, `try/except`, `throw` | return a `Result[T, E]`: `Ok(v)` or `Err(e)`; `r?` passes an Err up, `r.context("...")` adds where (§9) | | `panic!`, `assert`, `raise RuntimeError` | `panic(msg)`, `assert(cond, msg)`: a runtime error (§9) | | `if let Some(v) = x`, `x is not None` | `if x is Some(v):`, `not (x is None)` | | generics, templates, `` | `fn first[T](xs: [T]) T`, `struct Pair[A, B]` (§7, Generics) | | `any`, `interface{}`, `object` | a type parameter, an enum of the cases, or a function value | | `True`, `False` | `true`, `false` | | `&&`, `\|\|`, `!` | `and`, `or`, `not` | | `x++`, `x--` | `x += 1`, `x -= 1` | | `c ? a : b` | `a if c else b` | | `//`, `**` | `/` on ints; `math.pow(float, float)` | | `range(n)`, `range(a, b)`, `enumerate(xs)` | `0..n`, `a..b`, `for i, x in xs:` | | `xs.append(v)`, `xs.length`, `xs.size()` | `xs.push(v)`, `len(xs)` | | `xs.sort()`, `xs.reverse()` (in place) | `xs = xs.sorted()`, `xs = xs.reversed()` (`sort_by(fn(x) key)` for another order) | | `sum(xs)`, `zip(xs, ys)` | `sum(xs)` works; for zip, `for i in 0..len(xs):` and read `xs[i]`, `ys[i]` | | `[f(x) for x in xs if p(x)]` | `xs.filter(fn(x) p(x)).map(fn(x) f(x))` | | `lambda x: x + 1`, `x => x + 1`, `\|x\| x + 1` | `fn(x) x + 1` | | `int("42")`, `parseInt`, `strconv.Atoi` | `"42".to_int(0)` | | `str(xs)`, `toString()` of a struct | `json.encode(xs)`, or `print(xs)` | | `f"{x:.2f}"`, `%` formatting, `.format()` | `f"{x}"`; `f"{fmt.fixed(x, 2)}"`, `fmt.zero_pad`, `fmt.pad_left` | | `'single quotes'` | `"double quotes"`; `"""..."""` over several lines; `` `raw` `` with no escapes | | `else if`, `switch`, `case` | `elif`, `match` | | `do { } while`, `loop` | `while true:` + `break` | | `print(x, end="")` | build the whole line in a string, then `print` it | | `import std.math`, `use`, `#include` | `import math` | ## 20. Builtin names These are the builtin functions: ```text print len push pop chr copy float int sqrt abs sum min max str join split find contains starts_with ends_with replace trim upper lower to_int to_float byte repeat get remove keys values is_null string_at map filter sort_by reversed url_decode json_quote panic assert to_bytes to_str reserve resize_uninit ``` A function of your own, in the program or in a module, may have one of these names. A call `get(a, ...)` or `a.get(...)` goes to your `get` when its first parameter fits `a` and the number of arguments matches; otherwise it goes to the builtin, so the builtin still works on strings, arrays, and maps next to yours: ```tern struct Store: names: [str] fn get(s: Store, i: int) str: return s.names[i] fn len(s: Store) int: return len(s.names) fn main(): st = Store(["ada", "bob"]) ages = {"ada": 36} print(st.get(1), get(st, 0), len(st), ages.get("ada", 0), len("hey")) ``` ```output bob ada 2 36 3 ``` When the first argument is itself of a builtin type (a string, number, array, or map), a call that doesn't fit your function goes to the builtin; for any other type, the error is about your function. Functions in an `extern` block and `export fn` functions can't take a builtin's name, because C sees that name: rename an extern one with `= "c_name"`. ## 21. Idioms ```tern struct Parsed: ok: bool value: int fn parse_port(s: str) Parsed: # failure without exceptions n = s.to_int(-1) if n < 1 or n > 65535: return Parsed(false, 0) return Parsed(true, n) fn index_of(xs: [str], want: str) int: # search for i, x in xs: if x == want: return i return -1 fn main(): p = parse_port("8080") if p.ok: print("port", p.value) print(parse_port("x").ok, index_of(["a", "b"], "b"), reversed([1, 2, 3]), sum([4, 5])) csv = "name,age\nada,36\nbob,25" total = 0 for i, line in csv.split("\n"): if i == 0: continue cols = line.split(",") total += cols[1].to_int(0) print("total age", total) stack: [int] = [] stack.push(1) stack.push(2) top = stack.pop() queue = [1, 2, 3] head = queue[0] queue = queue[1..] # pop from the front print(top, stack, head, queue) ``` ```output port 8080 false 1 [3, 2, 1] 9 total age 61 2 [1] 1 [2, 3] ``` Graphs and trees that need links between nodes use indexes into an array: ```tern struct Node: name: str edges: [int] # indexes into the nodes array fn main(): nodes = [Node("a", [1, 2]), Node("b", [2]), Node("c", [])] seen = [false] * len(nodes) todo = [0] order: [str] = [] while len(todo) > 0: n = todo.pop() if seen[n]: continue seen[n] = true order.push(nodes[n].name) for e in nodes[n].edges: todo.push(e) print(order) ``` ```output ["a", "c", "b"] ``` # Tern grammar The exact syntax accepted by the compiler (`compiler/syntax/lexer.tn`, `compiler/syntax/parser.tn`), in EBNF. `{ x }` means zero or more, `[ x ]` optional, `|` alternatives, quoted text is literal. For what the constructs mean, see [reference.md](reference.md). ## Tokens ```text NEWLINE end of a line (not inside ( ) [ ] { }, and not after a trailing `\`) INDENT a line indented more than the one before; DEDENT: less (must match an earlier level) (a tab counts as 4 spaces; blank lines and comment-only lines are ignored) comment "#" up to the end of the line NAME letter or "_", then letters, digits, "_" (not a keyword) INT digit { digit | "_" } e.g. 42 1_000_000 (at most 9223372036854775807) | "0" ("x"|"X") hexdigit { hexdigit | "_" } e.g. 0xff (hex, binary, octal: up to 64 bits; | "0" ("b"|"B") bindigit { bindigit | "_" } above the largest int they are negative) | "0" ("o"|"O") octdigit { octdigit | "_" } FLOAT INT [ "." digit { digit } ] [ exponent ] with a "." or an exponent: 3.14 1e9 2.5e-3 exponent ("e"|"E") [ "+" | "-" ] digit { digit } (not "1." or ".5") STR '"' { char | escape } '"' one line; escapes \n \t \r \0 \" \\ \xHH (a byte), \uHHHH and \u{H..} (UTF-8) | '"""' { char | newline | escape } '"""' over lines; drops a blank first and last line and the indentation the lines share | "`" { char | newline } "`" raw: every character as written, no escapes FSTR 'f"' { char | "{{" | "}}" | "{" expr "}" } '"' | 'f"""' ... '"""' an f-string over lines, like """ keywords and break const continue do elif else enum export extern false fn for from if import in mut not null or parallel pass return struct take true while soft match (a keyword only at the start of a statement followed by an expression), is (only after an expression) symbols -> == != <= >= .. += -= *= /= %= &= |= ^= <<= >>= >>>= << >> >>> ( ) [ ] { } , : . = + - * / % < > & | ^ ~ ? ``` ## Program ```text program = { import | struct | enum | alias | const | fn | "export" fn | extern | NEWLINE } EOF alias = "type" NAME "=" type NEWLINE ("type" is a soft keyword here) import = "import" NAME NEWLINE | "from" NAME "import" NAME { "," NAME } NEWLINE struct = "struct" NAME [ tparams ] ":" NEWLINE INDENT field_line { field_line } DEDENT field_line = NAME { "," NAME } ":" type NEWLINE enum = "enum" NAME [ tparams ] ":" NEWLINE INDENT variant { variant } DEDENT variant = NAME [ "(" [ NAME ":" type { "," NAME ":" type } ] ")" ] NEWLINE const = "const" NAME [ ":" type ] "=" expr NEWLINE (int, float, str, or bool, computed from literals, constants, operators) fn = "fn" NAME [ tparams ] "(" [ param { "," param } ] ")" [ [ "->" ] type ] block param = NAME ":" [ "mut" | "take" ] type tparams = "[" tparam { "," tparam } "]" (generic: fn first[T], struct Pair[A, B]) tparam = NAME [ ":" NAME { "+" NAME } ] (constraints: Eq, Ord, Hash) extern = "extern" [ STR ] { ( "source" | "link" ) STR } ":" NEWLINE INDENT { "fn" NAME "(" [ param { "," param } ] ")" [ type ] [ "=" STR ] NEWLINE } DEDENT ``` ## Types ```text type = "int" | "float" | "bool" | "str" | "byte" | "ptr" | NAME [ "." NAME ] [ "[" type { "," type } "]" ] (a struct or enum, maybe from a module: http.Request; with type arguments: Pair[int, str]) | "[" type "]" (array) | "{" type ":" type "}" (map; keys: int, str, bool, or arrays, structs, enums, tuples of those) | "(" type "," type { "," type } ")" (tuple) | "fn" "(" [ fparam { "," fparam } ] ")" [ type ] (function; no type = returns nothing) fparam = [ "mut" | "take" ] type ``` ## Statements ```text block = ":" ( NEWLINE INDENT stmt { stmt } DEDENT | simple NEWLINE ) stmt = if | while | for | parallel | match | "do" block | fn | simple NEWLINE if = "if" expr block { "elif" expr block } [ "else" block ] while = "while" expr block for = "for" NAME "in" expr ".." expr block (range: start inclusive, end exclusive) | "for" NAME [ "," NAME ] "in" expr block (array: element or index, element; map: key or key, value; str: bytes) parallel = "parallel" block (each statement is a task; `do` groups) | "parallel" "for" NAME "in" expr ".." expr block match = "match" expr ":" NEWLINE INDENT arm { arm } DEDENT arm = pattern { "|" pattern } block (alternatives bind no names) pattern = "_" | INT | FLOAT | STR | "-" INT | "true" | "false" | NAME { "." NAME } [ "(" [ NAME { "," NAME } ] ")" ] (variant; "_" skips a value) simple = "return" [ exprs ] | "break" | "continue" | "pass" | NAME ":" type "=" expr (typed declaration) | place "=" exprs (declaration or assignment) | target "," target { "," target } "=" exprs (unpacks a tuple) | place ( "+=" | "-=" | "*=" | "/=" | "%=" | "&=" | "|=" | "^=" | "<<=" | ">>=" | ">>>=" ) expr | expr (a call, for its effect) place = NAME { "." ( NAME | INT ) | "[" expr "]" } target = place | "_" exprs = expr { "," expr } (two or more make a tuple) ``` ## Expressions Lowest to highest precedence; all binary operators are left-associative. ```text expr = or_expr [ "if" or_expr "else" expr ] (conditional: only one side runs) or_expr = and_expr { "or" and_expr } and_expr = cmp { "and" cmp } cmp = bor { ( "==" | "!=" | "<" | "<=" | ">" | ">=" | "in" ) bor | "is" pattern } (`is` binds names only in if/while conditions) bor = bxor { "|" bxor } bxor = band { "^" band } band = shift { "&" shift } (binary `&`: only after an operand) shift = sum { ( "<<" | ">>" | ">>>" ) sum } sum = term { ( "+" | "-" ) term } term = unary { ( "*" | "/" | "%" ) unary } unary = "-" unary | "not" unary | "~" unary | "&" place | postfix postfix = primary { "." NAME [ args ] | "." INT | "[" index "]" | args | "?" } (t.0: a tuple's value; x?: unwrap an Option / Result or return early) index = expr | expr ".." [ expr ] | ".." [ expr ] | expr "," expr { "," expr } (type arguments: f[int, str](...), after a generic name) args = "(" [ arg { "," arg } ] ")" arg = expr | NAME "=" expr (named: only when building a struct/variant) primary = INT | FLOAT | STR | FSTR | "true" | "false" | "null" | NAME | "[" [ expr { "," expr } ] "]" ([v] * n repeats v n times) | "{" [ expr ":" expr { "," expr ":" expr } ] "}" | "(" expr ")" | "(" expr "," expr { "," expr } [ "," ] ")" (a tuple) | "fn" "(" [ lparam { "," lparam } ] ")" expr (lambda: one expression) lparam = NAME [ ":" [ "mut" | "take" ] type ] ``` Notes: - `not` binds tighter than `in` and `==`, so write `not (k in m)` and `not (a == b)`. - `Name(...)` where `Name` is a struct builds a struct; `Enum.Variant(...)` builds a variant. - `x.f(args)` calls `f(x, args)` (or a function-valued field `f` of struct `x`). - `&` before an operand is the `mut`-argument marker (`f(&x)`); between two operands it is bitwise and. - There is no `c ? a : b` (write `a if c else b`), no `**`, no `//`, no `not in`, no `is not`. # Standard library source (library/std/) The comments at the top of each module document it. ## library/std/binary.tn ```tern # binary: numbers in binary formats: fixed-width integers and floats in big- or little-endian # byte order, and varints (as protobuf, Go's encoding/binary and SQLite's records write them). # Bytes are a str (any bytes; a [byte] converts with to_str(), no copy). Write with a Writer, # read in order with a Reader, or at any offset. # # import binary # w = binary.writer() # w.put_u16be(0xCAFE) # CA FE # w.put_u32le(7) # 07 00 00 00 # w.put_uvarint(300) # AC 02 # w.put_bytes("hi") # out = w.bytes() # the 10 bytes written # binary.set_u16be(&out, 0, len(out)) # overwrite 2 bytes at 0 # n = binary.u32le(out, 2) # read at an offset: 7 # # r = binary.reader(out) # read in order, checking as it goes # size = r.read_u16be() # kind = r.read_u32le() # x = r.read_uvarint() # name = r.read_bytes(2) # if r.err != "": # past the end, or a bad varint: every # return Err(r.err) # read after that gives 0 / "" # # Writer (bytes appended to the end; a put of an int writes its low bytes, so signed and # unsigned are alike: put_u32be(-1) is FF FF FF FF): # binary.writer() Writer # w.put_u8(v int) put_u16be put_u16le put_u32be put_u32le put_u64be put_u64le # w.put_f32be(v float) put_f32le put_f64be put_f64le # w.put_uvarint(v int) 7 bits a byte, low first (LEB128); v is read as unsigned (-1: 10 bytes) # w.put_varint(v int) zigzag first (0 -1 1 -2 -> 0 1 2 3): small negatives stay short # w.put_bytes(s str) w.len() int w.bytes() str (what was written; w can go on) # w.patch_u32be(i int, v int) overwrite 4 bytes written earlier (a length, once known) # Reader (data and a position; a read that doesn't fit sets err, the first problem with its # offset, moves to the end and returns 0 or "", so a parser checks once at the end): # binary.reader(data str) Reader fields: data str, pos int, err str # r.read_u8() int read_i8 read_u16be ... read_u64le, read_i16be ... read_i32le # r.read_f32be() float read_f32le read_f64be read_f64le # r.read_uvarint() int r.read_varint() int (err past 10 bytes or 64 bits, as Go's) # r.read_bytes(n int) str r.skip(n int) r.left() int (bytes after pos) # At an offset (a runtime error past the end of s, like s.byte(i)): # binary.u8(s, i) int binary.i8(s, i) int # binary.u16be(s, i) int u16le i16be i16le # binary.u32be(s, i) int u32le i32be i32le # binary.u64be(s, i) int u64le (int is 64-bit: a u64 above 2^63-1 reads negative) # binary.f32be(s, i) float f32le f64be f64le # binary.set_u8(s: mut str, i int, v int) set_u16be ... set_u64le, set_f32be ... set_f64le # Sizes: binary.uvarint_len(v int) int binary.varint_len(v int) int (bytes a put takes) # Many floats at once (vectors, embeddings, model weights), little-endian, one copy: # binary.f32s_le(s, i, n) [f32] n f32s from byte i (a runtime error past the end) # binary.f16s_le(s, i, n) [f32] n IEEE halves (2 bytes each) from byte i, widened # binary.f32s_le_bytes(xs [f32]) str 4 bytes each # binary.f16s_le_bytes(xs [f32]) str 2 bytes each: rounded to the nearest half (inf past 65504) const _BE = 16 const _S = 32 struct Writer: _buf: str # (its length is kept equal to _n; runtime/binary.h) _n: int _end: int # the buffer's capacity struct Reader: data: str pos: int _fail: int # 0, or the first failure: 1 + the bytes missing; -1 a varint cut short, # -2 one over 64 bits, -3 a negative count _at: int # where # ---------------------------------------------------------------- at an offset fn u8(s: str, i: int) int: return _get(s, i, 1) fn i8(s: str, i: int) int: return _get(s, i, 1 + _S) fn u16be(s: str, i: int) int: return _get(s, i, 2 + _BE) fn u16le(s: str, i: int) int: return _get(s, i, 2) fn i16be(s: str, i: int) int: return _get(s, i, 2 + _BE + _S) fn i16le(s: str, i: int) int: return _get(s, i, 2 + _S) fn u32be(s: str, i: int) int: return _get(s, i, 4 + _BE) fn u32le(s: str, i: int) int: return _get(s, i, 4) fn i32be(s: str, i: int) int: return _get(s, i, 4 + _BE + _S) fn i32le(s: str, i: int) int: return _get(s, i, 4 + _S) fn u64be(s: str, i: int) int: return _get(s, i, 8 + _BE) fn u64le(s: str, i: int) int: return _get(s, i, 8) fn f32be(s: str, i: int) float: return _getf(s, i, 4 + _BE) fn f32le(s: str, i: int) float: return _getf(s, i, 4) fn f64be(s: str, i: int) float: return _getf(s, i, 8 + _BE) fn f64le(s: str, i: int) float: return _getf(s, i, 8) # ---------------------------------------------------------------- many floats at once fn f32s_le(s: str, i: int, n: int) [f32]: return _f32s(s, i, n, 4) fn f16s_le(s: str, i: int, n: int) [f32]: return _f32s(s, i, n, 2) fn f32s_le_bytes(xs: [f32]) str: return _f32s_bytes(xs, 4) fn f16s_le_bytes(xs: [f32]) str: return _f32s_bytes(xs, 2) fn set_u8(s: mut str, i: int, v: int): _set(&s, i, v, 1) fn set_u16be(s: mut str, i: int, v: int): _set(&s, i, v, 2 + _BE) fn set_u16le(s: mut str, i: int, v: int): _set(&s, i, v, 2) fn set_u32be(s: mut str, i: int, v: int): _set(&s, i, v, 4 + _BE) fn set_u32le(s: mut str, i: int, v: int): _set(&s, i, v, 4) fn set_u64be(s: mut str, i: int, v: int): _set(&s, i, v, 8 + _BE) fn set_u64le(s: mut str, i: int, v: int): _set(&s, i, v, 8) fn set_f32be(s: mut str, i: int, v: float): _setf(&s, i, v, 4 + _BE) fn set_f32le(s: mut str, i: int, v: float): _setf(&s, i, v, 4) fn set_f64be(s: mut str, i: int, v: float): _setf(&s, i, v, 8 + _BE) fn set_f64le(s: mut str, i: int, v: float): _setf(&s, i, v, 8) fn uvarint_len(v: int) int: return _uvarint_len(v) fn varint_len(v: int) int: return _uvarint_len((v << 1) ^ (v >> 63)) # ---------------------------------------------------------------- Writer # Each put: room for the bytes (runtime/binary.h: _wfits, else _wroom grows or copies the # buffer), then the store. fn writer() Writer: return Writer("", 0, 0) fn len(w: Writer) int: return w._n fn bytes(w: mut Writer) str: return _wdone(&w._buf, &w._n, &w._end) # Overwrites 4 bytes already written at i (a length known only afterwards, say) fn patch_u32be(w: mut Writer, i: int, v: int): _set(&w._buf, i, v, 4 + _BE) fn put_bytes(w: mut Writer, s: str): if not _wfits(w._buf, w._n, w._end, len(s)): _wroom(&w._buf, w._n, &w._end, len(s)) _wbytes(w._buf, &w._n, s) fn put_u8(w: mut Writer, v: int): if not _wfits(w._buf, w._n, w._end, 8): _wroom(&w._buf, w._n, &w._end, 8) _wput(w._buf, &w._n, v, 1) fn put_u16be(w: mut Writer, v: int): if not _wfits(w._buf, w._n, w._end, 8): _wroom(&w._buf, w._n, &w._end, 8) _wput(w._buf, &w._n, v, 2 + _BE) fn put_u16le(w: mut Writer, v: int): if not _wfits(w._buf, w._n, w._end, 8): _wroom(&w._buf, w._n, &w._end, 8) _wput(w._buf, &w._n, v, 2) fn put_u32be(w: mut Writer, v: int): if not _wfits(w._buf, w._n, w._end, 8): _wroom(&w._buf, w._n, &w._end, 8) _wput(w._buf, &w._n, v, 4 + _BE) fn put_u32le(w: mut Writer, v: int): if not _wfits(w._buf, w._n, w._end, 8): _wroom(&w._buf, w._n, &w._end, 8) _wput(w._buf, &w._n, v, 4) fn put_u64be(w: mut Writer, v: int): if not _wfits(w._buf, w._n, w._end, 8): _wroom(&w._buf, w._n, &w._end, 8) _wput(w._buf, &w._n, v, 8 + _BE) fn put_u64le(w: mut Writer, v: int): if not _wfits(w._buf, w._n, w._end, 8): _wroom(&w._buf, w._n, &w._end, 8) _wput(w._buf, &w._n, v, 8) fn put_f32be(w: mut Writer, v: float): if not _wfits(w._buf, w._n, w._end, 8): _wroom(&w._buf, w._n, &w._end, 8) _wputf(w._buf, &w._n, v, 4 + _BE) fn put_f32le(w: mut Writer, v: float): if not _wfits(w._buf, w._n, w._end, 8): _wroom(&w._buf, w._n, &w._end, 8) _wputf(w._buf, &w._n, v, 4) fn put_f64be(w: mut Writer, v: float): if not _wfits(w._buf, w._n, w._end, 8): _wroom(&w._buf, w._n, &w._end, 8) _wputf(w._buf, &w._n, v, 8 + _BE) fn put_f64le(w: mut Writer, v: float): if not _wfits(w._buf, w._n, w._end, 8): _wroom(&w._buf, w._n, &w._end, 8) _wputf(w._buf, &w._n, v, 8) fn put_uvarint(w: mut Writer, v: int): if not _wfits(w._buf, w._n, w._end, 16): _wroom(&w._buf, w._n, &w._end, 16) _wuvarint(w._buf, &w._n, v) fn put_varint(w: mut Writer, v: int): put_uvarint(&w, (v << 1) ^ (v >> 63)) # ---------------------------------------------------------------- Reader fn reader(data: str) Reader: return Reader(data, 0, 0, 0) fn ok(r: Reader) bool: return r._fail == 0 fn error(r: Reader) str: if r._fail == 0: return "" return _fail_msg(len(r.data), r._fail, r._at) # true when n more bytes are there: one check before a fixed-size record, and the compiler # drops the reads' own checks after it fn has(r: Reader, n: int) bool: return _has(len(r.data), r.pos, n) fn left(r: Reader) int: if r.pos < 0 or r.pos > len(r.data): return 0 return len(r.data) - r.pos # the position of the next n bytes, moving past them; -1 when they aren't there (the first # failure is kept as numbers: nothing on the way calls a function, so a loop of reads keeps # the Reader in registers) fn _next(r: mut Reader, n: int) int: at = r.pos p = _take(&r.pos, len(r.data), n) if p < 0: _failed(&r, n + 1 if n >= 0 else -3, at) return p fn _failed(r: mut Reader, code: int, at: int): if r._fail == 0: r._fail = code r._at = at fn skip(r: mut Reader, n: int): _next(&r, n) fn read_bytes(r: mut Reader, n: int) str: p = _next(&r, n) if p < 0: return "" return r.data[p..p + n] fn read_u8(r: mut Reader) int: p = _next(&r, 1) return 0 if p < 0 else _load(r.data, p, 1) fn read_i8(r: mut Reader) int: p = _next(&r, 1) return 0 if p < 0 else _load(r.data, p, 1 + _S) fn read_u16be(r: mut Reader) int: p = _next(&r, 2) return 0 if p < 0 else _load(r.data, p, 2 + _BE) fn read_u16le(r: mut Reader) int: p = _next(&r, 2) return 0 if p < 0 else _load(r.data, p, 2) fn read_i16be(r: mut Reader) int: p = _next(&r, 2) return 0 if p < 0 else _load(r.data, p, 2 + _BE + _S) fn read_i16le(r: mut Reader) int: p = _next(&r, 2) return 0 if p < 0 else _load(r.data, p, 2 + _S) fn read_u32be(r: mut Reader) int: p = _next(&r, 4) return 0 if p < 0 else _load(r.data, p, 4 + _BE) fn read_u32le(r: mut Reader) int: p = _next(&r, 4) return 0 if p < 0 else _load(r.data, p, 4) fn read_i32be(r: mut Reader) int: p = _next(&r, 4) return 0 if p < 0 else _load(r.data, p, 4 + _BE + _S) fn read_i32le(r: mut Reader) int: p = _next(&r, 4) return 0 if p < 0 else _load(r.data, p, 4 + _S) fn read_u64be(r: mut Reader) int: p = _next(&r, 8) return 0 if p < 0 else _load(r.data, p, 8 + _BE) fn read_u64le(r: mut Reader) int: p = _next(&r, 8) return 0 if p < 0 else _load(r.data, p, 8) fn read_f32be(r: mut Reader) float: p = _next(&r, 4) return 0.0 if p < 0 else _loadf(r.data, p, 4 + _BE) fn read_f32le(r: mut Reader) float: p = _next(&r, 4) return 0.0 if p < 0 else _loadf(r.data, p, 4) fn read_f64be(r: mut Reader) float: p = _next(&r, 8) return 0.0 if p < 0 else _loadf(r.data, p, 8 + _BE) fn read_f64le(r: mut Reader) float: p = _next(&r, 8) return 0.0 if p < 0 else _loadf(r.data, p, 8) fn read_uvarint(r: mut Reader) int: p = r.pos v = _uvarint(r.data, &r.pos) if r.pos < 0: code = r.pos _failed(&r, code, p) r.pos = len(r.data) + 1 return v fn read_varint(r: mut Reader) int: p = r.pos v = _varint(r.data, &r.pos) if r.pos < 0: code = r.pos _failed(&r, code, p) r.pos = len(r.data) + 1 return v ``` ## library/std/bitset.tn ```tern # bitset: a set of small non-negative ints, one bit each (64 to an int): sieves, visited # sets, flags. Setting a bit past the end grows it. The functions that change one take it # `mut`, so bs.set(i) is set(&bs, i). # # import bitset # bs = bitset.new(1000) # bs.set(42) # bs.has(42) bs.count() bs.ones() # true 1 [42] # # bitset.new(n) Bitset bs.set(i) bs.unset(i) bs.toggle(i) bs.has(i) bool # bs.count() int (bits set) bs.ones() [int] (the set bits, ascending) bs.size() int # bs.union_with(o) bs.intersect_with(o) bs.subtract(o) (change bs; o is another Bitset) # # Counting and listing use the CPU's popcount and count-trailing-zeros (bitset._ones, _tz). struct Bitset: words: [int] fn new(n: int) Bitset: return Bitset([0] * ((max(n, 0) + 63) / 64)) fn size(b: Bitset) int: return len(b.words) * 64 fn set(b: mut Bitset, i: int): w = i >> 6 while w >= len(b.words): b.words.push(0) b.words[w] |= 1 << (i & 63) fn unset(b: mut Bitset, i: int): w = i >> 6 if i >= 0 and w < len(b.words): b.words[w] &= ~(1 << (i & 63)) fn toggle(b: mut Bitset, i: int): w = i >> 6 while w >= len(b.words): b.words.push(0) b.words[w] ^= 1 << (i & 63) fn has(b: Bitset, i: int) bool: w = i >> 6 if i < 0 or w >= len(b.words): return false return (b.words[w] >> (i & 63)) & 1 == 1 fn count(b: Bitset) int: n = 0 for x in b.words: n += _ones(x) return n fn ones(b: Bitset) [int]: out: [int] = [] for w, x in b.words: v = x while v != 0: out.push(w * 64 + _tz(v)) v &= v - 1 return out fn union_with(b: mut Bitset, o: Bitset): while len(b.words) < len(o.words): b.words.push(0) for k, x in o.words: b.words[k] |= x fn intersect_with(b: mut Bitset, o: Bitset): for k in 0..len(b.words): if k < len(o.words): b.words[k] &= o.words[k] else: b.words[k] = 0 fn subtract(b: mut Bitset, o: Bitset): for k in 0..min(len(b.words), len(o.words)): b.words[k] &= ~o.words[k] ``` ## library/std/compress/compress.tn ```tern # compress: zstd, gzip, zlib and raw deflate, of bytes (in a str: any bytes, NULs included). # Built in: zstd and libdeflate (runtime/vendor/) are linked in only when you import compress. # # import compress # packed = compress.zstd(text, 3) # one zstd frame # text = compress.unzstd(packed)? # Result[str, str] # body = compress.unzstd_max(req.body, 10_000_000)? # untrusted input: a size limit # gz = compress.gzip(text, 6) # a .gz file's bytes # text = compress.gunzip(fs.read("log.gz"))? # every member, as gzip -d does # # compress.zstd(data str, level int) str a zstd frame holding data, its size in the # header. Levels 1..19 (higher: smaller, slower; # 3 is zstd's default), up to 22, or negative for # faster and bigger; out of range is clamped # compress.unzstd(data str) Result[str, str] the bytes of every zstd frame in data, in # order (frames with or without their size, e.g. # from `zstd` streaming or another library); # Err when data isn't whole zstd frames # compress.unzstd_max(data str, max_out int) Result[str, str] # the same, Err("compress: unzstd: too large") # once the output passes max_out bytes, before # it is all in memory (for untrusted input: a # small frame can hold gigabytes) # compress.zstd_bound(n int) int the most bytes zstd of n bytes can take # # compress.gzip(data str, level int) str one gzip member (a .gz file): levels 1..12 # (6 is gzip's default; 10-12 are slow and # smallest; 0 stores), out of range is clamped # compress.gunzip(data str) Result[str, str] every member in data, in order (`cat a.gz b.gz` # is one valid .gz); Err when data isn't gzip, is # cut short, or a checksum is wrong # compress.gunzip_max(data str, max_out int) Result[str, str] with a size limit # compress.zlib(data, level) str compress.unzlib(data) / unzlib_max(data, max_out) # a zlib stream (HTTP's "deflate", PNG's chunks) # compress.deflate(data, level) str compress.inflate(data) / inflate_max(data, max_out) # raw deflate, no header or checksum (zip entries) # # A str holds up to 4GB, so that is the most either side can be. In a task, a call runs on # the task's thread (a big input holds up that core's other tasks briefly; use task.blocking # for very large ones). fn _err() str: e = error() if e.starts_with("compress."): return "compress: " + e[9..] return "compress: " + e fn unzstd(data: str) Result[str, str]: out = _unzstd(data, -1) if error() != "": return Err(_err()) return Ok(out) fn unzstd_max(data: str, max_out: int) Result[str, str]: if max_out < 0: return Err("compress: unzstd: max_out is negative") out = _unzstd(data, max_out) if error() != "": return Err(_err()) return Ok(out) ``` ## library/std/compress/deflate.tn ```tern # compress: gzip, zlib and raw deflate (runtime/vendor/libdeflate; see compress.tn's top). const _RAW = 0 const _ZLIB = 1 const _GZIP = 2 fn gzip(data: str, level: int) str: return _deflate(data, level, _GZIP) fn gunzip(data: str) Result[str, str]: return _un(data, -1, _GZIP, "gunzip") fn gunzip_max(data: str, max_out: int) Result[str, str]: return _un(data, max_out, _GZIP, "gunzip") fn zlib(data: str, level: int) str: return _deflate(data, level, _ZLIB) fn unzlib(data: str) Result[str, str]: return _un(data, -1, _ZLIB, "unzlib") fn unzlib_max(data: str, max_out: int) Result[str, str]: return _un(data, max_out, _ZLIB, "unzlib") fn deflate(data: str, level: int) str: return _deflate(data, level, _RAW) fn inflate(data: str) Result[str, str]: return _un(data, -1, _RAW, "inflate") fn inflate_max(data: str, max_out: int) Result[str, str]: return _un(data, max_out, _RAW, "inflate") fn _un(data: str, max_out: int, kind: int, name: str) Result[str, str]: if max_out < -1: return Err(f"compress: {name}: max_out is negative") out = _inflate(data, max_out, kind) if error() != "": return Err(_err()) return Ok(out) ``` ## library/std/crypto.tn ```tern # crypto: hashes, MACs, password hashing, secure random numbers, hex / base64 / UUIDs. # Self-contained C (runtime/crypto.h); nothing is linked in unless you import crypto. # # import crypto # digest = crypto.hex(crypto.sha256("hello")) # "2cf24dba5fb0a30e..." # mac = crypto.hmac_sha256(secret, body) # 32 raw bytes # ok = crypto.equal(mac, crypto.unhex(req.header("x-signature"))) # constant time # h = crypto.hash_password(pw) # "$argon2id$v=19$m=19456,t=2,p=1$..." # if crypto.verify_password(attempt, h): ... # token = crypto.base64url(crypto.random_bytes(32)) # a URL-safe session token # id = crypto.uuid7() # time-ordered, good for primary keys # # Hashes and MACs return raw bytes in a str (crypto.hex / crypto.base64 make them printable): # crypto.sha256(data str) str (32 bytes) crypto.sha512(data str) str (64 bytes) # crypto.hmac_sha256(key str, msg str) str crypto.hmac_sha512(key str, msg str) str # crypto.sha1(data str) str (20 bytes) crypto.hmac_sha1(key str, msg str) str # (SHA-1 is broken for signatures and collisions; it's here # for TOTP codes and old protocols that require it) # Keys from passwords: # crypto.hash_password(pw str) str Argon2id, a random salt, OWASP's cost (19 MiB, t=2, p=1), # as a PHC string (the format other argon2 libraries read) # crypto.verify_password(pw str, hash str) bool (false for a malformed hash too) # crypto.argon2id(pw str, salt str, t int, m_kib int, p int, length int) str raw Argon2id # crypto.pbkdf2_sha256(pw str, salt str, iterations int, length int) str # From a task, these run on the blocking-thread pool (like task.blocking), so the other # tasks keep running while a password is hashed. # Randomness, from the operating system (arc4random / getrandom / WASI random_get): # crypto.random_bytes(n int) str crypto.random_int(lo int, hi int) int (lo..hi inclusive) # crypto.uuid4() str crypto.uuid7() str ("0192f7e4-8c1a-7b3e-9f2d-...") # Encryption (ChaCha20-Poly1305, RFC 8439: authenticated, so a changed byte is caught): # sealed = crypto.seal(key, plaintext) str key: 32 random bytes (crypto.random_bytes(32)); # a fresh random nonce each time, kept in the output # (12 bytes + the ciphertext + a 16-byte tag) # crypto.unseal(key, sealed) str the plaintext; "" and crypto.error() if the data was # changed or the key is wrong (try_unseal: a Result) # crypto.chacha20_poly1305(key, nonce, aad, plaintext) str the raw AEAD: you choose the # crypto.chacha20_poly1305_open(key, nonce, aad, sealed) str 12-byte nonce (never reuse one # with the same key) and extra authenticated data # Signatures (checking them; written in Tern, so nothing more is linked): # crypto.rsa_verify_sha256(n str, e str, msg str, sig str) bool # RSASSA-PKCS1-v1_5 with SHA-256, JWT's RS256: n and e are the # public key's big-endian bytes (a JWK's "n" and "e" after # base64url_decode); 1024- to 8192-bit keys, e up to 8 bytes. # The expected block is built and compared whole. # Comparing secrets: # crypto.equal(a str, b str) bool takes the same time wherever a and b differ # Encodings: # crypto.hex(data str) str crypto.unhex(s str) str # crypto.base64(data str) str crypto.base64_decode(s str) str (with = padding) # crypto.base64url(data str) str crypto.base64url_decode(s str) str (- and _, no padding) # Decoders return "" for bad input and crypto.error() says why ("" after a success). # crypto.error() str # with Result: bytes = crypto.try_base64_decode(s)? (Err: "crypto: base64_decode: not base64") fn _err() str: e = error() if e.starts_with("crypto."): return "crypto: " + e[7..] return "crypto: " + e fn try_unhex(s: str) Result[str, str]: out = unhex(s) if error() != "": return Err(_err()) return Ok(out) fn try_base64_decode(s: str) Result[str, str]: out = base64_decode(s) if error() != "": return Err(_err()) return Ok(out) fn try_base64url_decode(s: str) Result[str, str]: out = base64url_decode(s) if error() != "": return Err(_err()) return Ok(out) fn seal(key: str, plaintext: str) str: nonce = random_bytes(12) out = chacha20_poly1305(key, nonce, "", plaintext) if error() != "": return "" return nonce + out fn unseal(key: str, sealed: str) str: if len(sealed) < 28: # (the C side says why: too short) return chacha20_poly1305_open(key, "\x00" * 12, "", "") return chacha20_poly1305_open(key, sealed[..12], "", sealed[12..]) # with Result: secret = crypto.try_unseal(key, sealed)? (Err: "crypto: open: the data was changed, or the key is wrong") fn try_unseal(key: str, sealed: str) Result[str, str]: out = unseal(key, sealed) if error() != "": return Err(_err()) return Ok(out) # ---- RSA signature verification (RSASSA-PKCS1-v1_5 with SHA-256, as JWT's RS256) ---- # Numbers are little-endian arrays of 24-bit limbs, so a product of two fits in 48 bits and a # column of up to 2^15 of them in an int without carrying. const _LIMB = 24 const _MASK = 16777215 # 2^24 - 1 fn _from_bytes(be: str) [int]: out: [int] = [] acc = 0 bits = 0 i = len(be) - 1 while i >= 0: acc = acc | (be.byte(i) << bits) bits += 8 if bits >= _LIMB: out.push(acc & _MASK) acc = acc >> _LIMB bits -= _LIMB i -= 1 if bits > 0: out.push(acc) return _trim(out) fn _trim(a: [int]) [int]: n = len(a) while n > 0 and a[n - 1] == 0: n -= 1 return a[..n] # The low `size` bytes, big-endian. fn _to_bytes(a: [int], size: int) str: out = "" for i in 0..size: k = size - 1 - i # byte k from the bottom bit = k * 8 limb = bit / _LIMB v = 0 if limb < len(a): v = a[limb] >> (bit % _LIMB) if bit % _LIMB > _LIMB - 8 and limb + 1 < len(a): v = v | (a[limb + 1] << (_LIMB - bit % _LIMB)) out += chr(v & 255) return out fn _cmp(a: [int], b: [int]) int: x = _trim(a) y = _trim(b) if len(x) != len(y): return len(x) - len(y) i = len(x) - 1 while i >= 0: if x[i] != y[i]: return x[i] - y[i] i -= 1 return 0 fn _mul(a: [int], b: [int]) [int]: out: [int] = [] for i in 0..len(a) + len(b) + 1: out.push(0) for i, x in a: if x == 0: continue for j, y in b: out[i + j] += x * y # carry this row before the columns can overflow c = 0 for k in i..len(out): v = out[k] + c out[k] = v & _MASK c = v >> _LIMB if c == 0 and k >= i + len(b): break return _trim(out) # a mod n, one bit at a time from the top (r stays below 2n, so one subtraction a step). fn _mod(a: [int], n: [int]) [int]: if _cmp(a, n) < 0: return a r: [int] = [] for i in 0..len(n) + 1: r.push(0) bit = len(a) * _LIMB - 1 while bit >= 0: # r = 2r + the next bit of a c = (a[bit / _LIMB] >> (bit % _LIMB)) & 1 for k in 0..len(r): v = (r[k] << 1) | c r[k] = v & _MASK c = v >> _LIMB if _cmp(r, n) >= 0: borrow = 0 for k in 0..len(r): m = 0 if k < len(n): m = n[k] v = r[k] - m - borrow if v < 0: v += _MASK + 1 borrow = 1 else: borrow = 0 r[k] = v bit -= 1 return _trim(r) fn _powmod(s: [int], e: str, n: [int]) [int]: acc: [int] = [1] started = false for i in 0..len(e): b = e.byte(i) for j in 0..8: if started: acc = _mod(_mul(acc, acc), n) if (b >> (7 - j)) & 1 == 1: acc = _mod(_mul(acc, s), n) started = true return acc # DER of DigestInfo { sha256, NULL } before the 32-byte hash (RFC 8017, 9.2 note 1). const _SHA256_INFO = "3031300d060960864801650304020105000420" # Whether sig is msg signed with the RSA key (n, e: big-endian bytes, as a JWK's "n" and "e" # after base64url_decode), PKCS#1 v1.5 with SHA-256 (JWT's RS256). The expected block is built # and compared whole, never parsed out of the signature. fn rsa_verify_sha256(n: str, e: str, msg: str, sig: str) bool: nn = _from_bytes(n) k = (len(nn) * _LIMB + 7) / 8 while k > 0 and _to_bytes(nn, k).byte(0) == 0: k -= 1 if k < 128 or k > 1024 or len(sig) != k or len(e) == 0 or len(e) > 8: return false # 1024 to 8192-bit keys; small public exponents s = _from_bytes(sig) if _cmp(s, nn) >= 0: return false t = unhex(_SHA256_INFO) + sha256(msg) em = chr(0) + chr(1) for i in 0..k - 3 - len(t): em += chr(255) em += chr(0) + t return equal(_to_bytes(_powmod(s, e, nn), k), em) ``` ## library/std/csv.tn ```tern # csv: reading and writing comma-separated values (RFC 4180, like Go's encoding/csv). # # import csv # rows = csv.parse(text)? # Result[[[str]], str]: every record, as fields # people = csv.records(text)? # Result[[{str: str}], str]: the first row names # # the columns, each later row is {column: field} # out = csv.format(rows) # [[str]] back to CSV text, quoting where needed # line = csv.format_row(["a", "b,c"]) # one record and its line end: a,"b,c"\n # # A big file one record at a time, without an array of all of them: # r = csv.reader(text) # row: [str] = [] # while r.next(&row): # false at the end, or at an error # ... # if r.err != "": ... # why it stopped early ("" at a clean end) # # Options (csv.options() gives the defaults; change the fields you need): # o = csv.options() # o.sep = "\t" # the field separator, one byte ("," by default; csv.tsv() # # gives the options for tab-separated values) # o.comment = "#" # lines starting with it are skipped ("" by default: none) # o.fields = -1 # how many fields each record has: 0 (the default) means as # # many as the first, n exactly n, -1 any number # o.lazy_quotes = true # a " inside an unquoted field, or a lone " inside a quoted # # one, is kept as text instead of an error # o.trim_leading_space = true # spaces, tabs and \r before each field are dropped # o.crlf = true # format writes \r\n line ends (\n by default) # csv.parse_with(text, o) csv.records_with(text, o) csv.reader_with(text, o) # csv.format_with(rows, o) csv.format_row_with(row, o) # # Reading: a field in double quotes may hold separators, line breaks and "" (one quote). # \r\n line ends read as \n, also inside quoted fields. Empty lines are skipped, and a # UTF-8 byte order mark at the start is dropped. Errors name the line and column (1-based, # in bytes): `csv: line 3, column 7: bare " in a field without quotes`. # Writing: a field is quoted when it holds the separator, a quote, \r or \n, or starts with # a space or tab; quotes inside are doubled. Fields are text: convert numbers with str(n) and # parse them back with s.to_int(0) / s.to_float(0.0). struct Options: sep: str comment: str fields: int lazy_quotes: bool trim_leading_space: bool crlf: bool struct Reader: text: str pos: int line: int # the line the next record starts on (1-based) err: str # why next stopped early; "" while all is well opt: Options sep: int comment: int # the comment byte, or -1 want: int # the field count records must have: -1 any, 0 not known yet fn options() Options: return Options(",", "", 0, false, false, false) fn tsv() Options: return Options("\t", "", 0, false, false, false) fn reader(text: str) Reader: return reader_with(text, options()) fn reader_with(text: str, o: Options) Reader: pos = 0 if len(text) >= 3 and text.byte(0) == 0xef and text.byte(1) == 0xbb and text.byte(2) == 0xbf: pos = 3 r = Reader(text, pos, 1, "", o, 44, -1, o.fields) if len(o.sep) != 1 or o.sep == "\"" or o.sep == "\r" or o.sep == "\n": r.err = "csv: the separator must be one byte other than a quote or a line break" r.pos = len(text) return r r.sep = o.sep.byte(0) if len(o.comment) > 1 or o.comment == o.sep or o.comment == "\"": r.err = "csv: the comment marker must be one byte other than the separator or a quote" r.pos = len(text) return r if len(o.comment) == 1: r.comment = o.comment.byte(0) return r fn _fail(r: mut Reader, line: int, col: int, why: str) bool: r.err = f"csv: line {line}, column {col}: {why}" r.pos = len(r.text) return false # The next record into row (its old fields are dropped); false at the end of the text or at # an error (then r.err says why). fn next(r: mut Reader, row: mut [str]) bool: s = r.text n = len(s) sep = r.sep pos = r.pos line = r.line # skip empty lines and comments while pos < n: c = s.byte(pos) if c == 10: pos += 1 line += 1 elif c == 13 and pos + 1 < n and s.byte(pos + 1) == 10: pos += 2 line += 1 elif c == 13 and pos + 1 == n: # (a \r at the very end is dropped, as in Go) pos += 1 elif c == r.comment: nl = s.find_from("\n", pos) if nl < 0: pos = n else: pos = nl + 1 line += 1 else: break if pos >= n: r.pos = n r.line = line row = [] return false start_line = line # the usual options: the whole record at once (runtime/strings.h); -1 sends anything # unusual (a misplaced quote) to the code below, which says where or reads it lazily p = -1 if not r.opt.trim_leading_space: p = _record(s, pos, sep, &row, &line) if p >= 0: pos = min(p, n) else: line_start = pos # for column numbers in errors k = 0 # fields so far: row[k] = ... reuses row's strings lazy = r.opt.lazy_quotes while true: if r.opt.trim_leading_space: while pos < n and _space(s.byte(pos)): pos += 1 if pos < n and s.byte(pos) == 34: # a quoted field qcol = pos - line_start + 1 pos += 1 start = pos if k == len(row): row.push("") first = true # row[k] = the first piece, += each later one while true: q = s.find_from("\"", pos) if q < 0: if not r.opt.lazy_quotes: return _fail(&r, start_line, qcol, "a quoted field is never closed") end = n if end > start and s.byte(end - 1) == 13: # (dropped, as in Go) end -= 1 _piece(&row, k, first, s, start, end, &line, &line_start) pos = n break if q + 1 < n and s.byte(q + 1) == 34: _piece(&row, k, first, s, start, q + 1, &line, &line_start) # "" is " first = false pos = q + 2 start = pos continue _piece(&row, k, first, s, start, q, &line, &line_start) first = false pos = q + 1 if pos >= n: break c = s.byte(pos) if c == sep or c == 10 or (c == 13 and (pos + 1 == n or s.byte(pos + 1) == 10)): break if not r.opt.lazy_quotes: return _fail(&r, line, pos - line_start + 1, "text after the closing quote of a field (\"\" is a quote inside one)") row[k] += "\"" start = pos k += 1 else: # a field without quotes: up to the separator or the line end start = pos while pos < n: c = s.byte(pos) if c == sep or c == 10: break if c == 34 and not lazy: return _fail(&r, line, pos - line_start + 1, "bare \" in a field without quotes (quote the field and double the \")") pos += 1 end = pos if end > start and s.byte(end - 1) == 13 and (pos >= n or s.byte(pos) == 10): end -= 1 if k < len(row): row[k] = s[start..end] else: row.push(s[start..end]) k += 1 # after a field: a separator, a line end or the end of the text if pos >= n: break c = s.byte(pos) if c == sep: pos += 1 continue if c == 13: pos += 1 if pos < n: # the \n (a \r at the end of the text ends no line) pos += 1 line += 1 break while len(row) > k: row.pop() r.pos = pos r.line = line if r.want == 0: r.want = len(row) elif r.want > 0 and len(row) != r.want: why = f"the first record has {_fields(r.want)}" if r.opt.fields > 0: why = f"options.fields is {r.want}" r.err = f"csv: line {start_line}: {_fields(len(row))}, but {why}" r.pos = n return false return true fn _space(c: int) bool: # what trim_leading_space drops: ASCII space, except \n return c == 32 or c == 9 or c == 13 or c == 11 or c == 12 fn _fields(n: int) str: if n == 1: return "1 field" return f"{n} fields" # row[k] = s[a..b] (first) or row[k] += s[a..b], with \r\n made \n, counting its line breaks; # straight from s, into row[k]'s own string, when there are none (the usual case) fn _piece(row: mut [str], k: int, first: bool, s: str, a: int, b: int, line: mut int, line_start: mut int): if not s[a..b].contains("\n"): if first: row[k] = s[a..b] else: row[k] += s[a..b] return t = s[a..b] i = t.find("\n") while i >= 0: line += 1 line_start = a + i + 1 i = t.find_from("\n", i + 1) t = t.replace("\r\n", "\n") if first: row[k] = t else: row[k] += t fn parse(text: str) Result[[[str]], str]: return parse_with(text, options()) fn parse_with(text: str, o: Options) Result[[[str]], str]: r = reader_with(text, o) rows: [[str]] = [] row: [str] = [] while r.next(&row): rows.push(row) row = [] # (a new array: the pushed one is kept) if r.err != "": return Err(r.err) return Ok(rows) fn records(text: str) Result[[{str: str}], str]: return records_with(text, options()) fn records_with(text: str, o: Options) Result[[{str: str}], str]: r = reader_with(text, o) out: [{str: str}] = [] head: [str] = [] if not r.next(&head): if r.err != "": return Err(r.err) return Ok(out) row: [str] = [] while r.next(&row): m: {str: str} = {} for i, f in row: if i < len(head): m[head[i]] = f out.push(m) if r.err != "": return Err(r.err) return Ok(out) fn _needs_quotes(f: str, sep: int) bool: if f == "": return false c = f.byte(0) if c == 32 or c == 9: return true for i in 0..len(f): c = f.byte(i) if c == sep or c == 34 or c == 10 or c == 13: return true return false fn _put_row(out: mut str, row: [str], o: Options): if len(row) == 1 and row[0] == "": out += "\"\"" # not an empty line, which reading skips sep = o.sep.byte(0) for i, f in row: if i > 0: out += o.sep if _needs_quotes(f, sep): out += "\"" out += f.replace("\"", "\"\"") out += "\"" else: out += f if o.crlf: out += "\r\n" else: out += "\n" fn format_row(row: [str]) str: return format_row_with(row, options()) fn format_row_with(row: [str], o: Options) str: out = "" _put_row(&out, row, o) return out fn format(rows: [[str]]) str: return format_with(rows, options()) fn format_with(rows: [[str]], o: Options) str: out = "" for row in rows: _put_row(&out, row, o) return out ``` ## library/std/deque.tn ```tern # deque: a double-ended queue: push and pop at either end in O(1) (a queue for BFS, a # sliding window, undo history). A value like any other; the functions that change one take # it `mut`, so d.push_back(x) is push_back(&d, x). # # import deque # d = deque.new[str]() # d.push_back("b") # d.push_front("a") # d.pop_front() # Some("a") # # deque.new[T]() Deque[T] deque.of(xs: [T]) Deque[T] # d.push_back(x) d.push_front(x) d.pop_back() Option[T] d.pop_front() Option[T] # d.front() Option[T] d.back() Option[T] d.at(i) T (0 is the front) d.put(i, x) # d.len() d.is_empty() d.to_array() [T] (front to back) d.clear() # # A ring buffer whose size is a power of two (positions wrap with a mask). Growing doubles # it, filling the new half with a copy of the old (Tern has no empty slots); a slot that was # popped keeps its old value until it's written again. struct Deque[T]: buf: [T] head: int n: int fn new[T]() Deque[T]: return Deque[T]([], 0, 0) fn of[T](xs: [T]) Deque[T]: # capacity a power of two (filler copies after the values), so positions wrap with a mask buf = xs cap = 1 while cap < len(xs): cap *= 2 k = 0 while len(buf) < cap: buf.push(xs[k]) k += 1 return Deque[T](buf, 0, len(xs)) fn len[T](d: Deque[T]) int: return d.n fn is_empty[T](d: Deque[T]) bool: return d.n == 0 # room for one more: the ring doubles by appending a copy of itself. The values that had # wrapped around to the front then also sit right after the old end, so they follow on from # head without moving, and the array grows in place when it can (no second buffer). fn _grow[T](d: mut Deque[T]): cap = len(d.buf) if d.n < cap or cap == 0: return for k in 0..cap: d.buf.push(d.buf[k]) fn push_back[T](d: mut Deque[T], x: T): cap = len(d.buf) if d.n == cap: if cap == 0: d.buf.push(x) d.n = 1 return _grow(&d) cap = len(d.buf) d.buf[(d.head + d.n) & (cap - 1)] = x d.n += 1 fn push_front[T](d: mut Deque[T], x: T): if d.n == len(d.buf): if d.n == 0: d.buf.push(x) d.n = 1 return _grow(&d) d.head = (d.head - 1) & (len(d.buf) - 1) d.buf[d.head] = x d.n += 1 fn pop_front[T](d: mut Deque[T]) Option[T]: if d.n == 0: return None x = d.buf[d.head] d.head = (d.head + 1) & (len(d.buf) - 1) d.n -= 1 return Some(x) fn pop_back[T](d: mut Deque[T]) Option[T]: if d.n == 0: return None d.n -= 1 return Some(d.buf[(d.head + d.n) & (len(d.buf) - 1)]) fn front[T](d: Deque[T]) Option[T]: if d.n == 0: return None return Some(d.buf[d.head]) fn back[T](d: Deque[T]) Option[T]: if d.n == 0: return None return Some(d.buf[(d.head + d.n - 1) & (len(d.buf) - 1)]) fn _slot[T](d: Deque[T], i: int) int: if i < 0 or i >= d.n: panic(f"deque: index {i} is out of range (length {d.n})") return (d.head + i) & (len(d.buf) - 1) fn at[T](d: Deque[T], i: int) T: return d.buf[_slot(d, i)] fn put[T](d: mut Deque[T], i: int, x: T): d.buf[_slot(d, i)] = x fn to_array[T](d: Deque[T]) [T]: out: [T] = [] m = len(d.buf) - 1 for k in 0..d.n: out.push(d.buf[(d.head + k) & m]) return out fn clear[T](d: mut Deque[T]): d.buf = [] d.head = 0 d.n = 0 ``` ## library/std/flag/fields.tn ```tern # flag: a struct's fields, as the table the compiler makes for its type (flag._layout(v)), # one record per field: # [size: 2 bytes][n][name: n bytes][n][kind][L or -][O or -][index: 2 bytes][choices: "A,B"] # (kind i f b s e; L: an array; O: an Option; index: the field's place in the struct, for # flag._set; choices: an enum's variants in order; size counts the whole record), or "!" and # a message when a field can't be a flag. A parse looks up only the flags it's given # (flag._ref skips a record whose name length differs). struct Field: name: str # the struct field long: str # its flag, without "--" kind: str # "int" "float" "bool" "str" "enum" list: bool optional: bool choices: [str] # an enum's variants # The table of T's fields, or an Err saying which field can't be a flag. fn _table[T](v: T) Result[str, str]: t = _layout(v) if t != "" and t.byte(0) == 33: return Err(t[1..]) return Ok(t) # A field is the index of its kind letter in the table, or -1 (flag._ref: --min-len and # --min_len both name min_len). fn _field_ref(t: str, name: str) int: return _ref(t, name, 0, len(name)) fn _kind(t: str, r: int) int: return t.byte(r) fn _listed(t: str, r: int) bool: return t.byte(r + 1) == 76 fn _index(t: str, r: int) int: return t.byte(r + 3) * 256 + t.byte(r + 4) # where field r's record starts, and where its name starts fn _rec(t: str, r: int) int: return r - 4 - t.byte(r - 1) fn _name_at(t: str, r: int) int: return r - 1 - t.byte(r - 1) fn _rec_end(t: str, r: int) int: a = _rec(t, r) return a + t.byte(a) * 256 + t.byte(a + 1) fn _name(t: str, r: int) str: return t[_name_at(t, r)..r - 1] fn _choices(t: str, r: int) [str]: return t[r + 5.._rec_end(t, r)].split(",") fn _field(t: str, r: int) Field: name = _name(t, r) k = t.byte(r) kind = "int" if k == 105 else "float" if k == 102 else "bool" if k == 98 else "enum" if k == 101 else "str" f = Field(name, name.replace("_", "-"), kind, _listed(t, r), t.byte(r + 2) == 79, []) if f.kind == "enum": f.choices = _choices(t, r) return f # Every field, in order (for --help and suggestions). fn _fields(t: str) [Field]: out: [Field] = [] a = 0 while a + 3 <= len(t): out.push(_field(t, a + 4 + t.byte(a + 2))) a += t.byte(a) * 256 + t.byte(a + 1) return out ``` ## library/std/flag/flag.tn ```tern # flag: command-line flags into a struct. Each field is a flag (min_len is --min-len), its # type says what the flag takes, and the struct's values before parsing are the defaults. # # import flag # # enum Mode: # Fast # Safe # # struct Opts: # port: int # --port 8080, --port=8080, -p 8080, -p8080 # verbose: bool # --verbose, --verbose=false, --no-verbose, -v (and -vq) # tag: [str] # repeatable: --tag a --tag b # config: Option[str] # None unless given # mode: Mode # --mode safe (any case); help lists fast|safe # # fn main() Result[int, str]: # o = Opts(port=8080, verbose=false, tag=[], config=None, mode=Mode.Fast) # c = flag.cli("serve", "Serve files over HTTP.") # c.flag("port", "p", "port to listen on") # c.flag("verbose", "v", "log every request") # c.env("port", "PORT") # used when --port isn't given # c.args("[DIR]") # what comes after the flags, for the usage line # dirs = flag.parse(c, &o)? # the positional arguments # ... # # flag.cli(name, about) Cli name and one-line description for the help # c.flag(field, short, help) a one-letter short form ("" for none) and help # c.env(field, var) an environment variable the field falls back to # c.require(field) the flag must be given (or come from its env) # c.args(text) the usage line's positional part ("[FILE...]") # c.stop_at_args() flags end at the first positional (for commands: # "tool -v build --release" leaves build's flags) # flag.parse(c, &out) Result[[str], str] parse os.args() (without the program) into out # flag.parse_from(c, args, &out) Result[[str], str] the same for args you pass (tests, # a subcommand's arguments) # flag.usage(c, out) str the help text, with out's values as defaults # # -h / --help prints the usage and exits with status 0 (c.exit_on_help = false makes it an # Err holding the usage instead). Mistakes are an Err naming the flag: "unknown flag --prot # (did you mean --port?)", "--port: expected an integer, got `abc`", "--mode: expected one of # fast, safe". "--" ends the flags; "-" alone is a positional (stdin, by convention). # Field types: int, float, bool, str, an enum without payloads, an array of one of those # (repeatable), or an Option of one (None unless the flag is given). import os struct Cli: name: str about: str positional: str notes: [Note] # what c.flag and c.env said, one per field required: [str] stop_early: bool exit_on_help: bool struct Note: field: str short: str # its one-letter form ("" for none) help: str env: str # the environment variable it falls back to ("" for none) fn cli(name: str, about: str) Cli: return Cli(name, about, "", [], [], false, true) fn _note(c: mut Cli, field: str) int: for k, n in c.notes: if n.field == field: return k c.notes.push(Note(field, "", "", "")) return len(c.notes) - 1 fn flag(c: mut Cli, field: str, short: str, help: str): k = _note(&c, field) if short != "": c.notes[k].short = short c.notes[k].help = help fn env(c: mut Cli, field: str, var: str): k = _note(&c, field) c.notes[k].env = var fn require(c: mut Cli, field: str): c.required.push(field) fn args(c: mut Cli, text: str): c.positional = text fn stop_at_args(c: mut Cli): c.stop_early = true fn parse[T](c: Cli, out: mut T) Result[[str], str]: a = os.args() return parse_from(c, a[1..], &out) # Each flag is written straight into its field (flag._set: compiler-made glue per struct); # on an Err, or for --help, out goes back to what it was. fn parse_from[T](c: Cli, args: [str], out: mut T) Result[[str], str]: t = _table(out)? orig = out res = _parse(c, t, args, &out, orig) if res.is_err(): out = orig return res fn _parse[T](c: Cli, t: str, args: [str], out: mut T, orig: T) Result[[str], str]: seen = Seen(0, []) rest: [str] = [] i = 0 n = len(args) while i < n: a = args[i] i += 1 if len(a) < 2 or a.byte(0) != 45: rest.push(a) if c.stop_early: for k in i..n: rest.push(args[k]) break continue if a == "--": for k in i..n: rest.push(args[k]) break if a == "-h" or a == "--help": u = usage(c, orig) if c.exit_on_help: print(u) os.exit(0) return Err(u) # the field r takes value; the flag was a[..end], or note k's short form r = -1 k = -1 end = len(a) value = "" if a.byte(1) == 45: # --name, --name=value, --no-name eq = _byte_at(a, 61, 2) if eq == len(a): eq = -1 if eq >= 0: end = eq r = _ref(t, a, 2, end) if r < 0 and eq < 0 and a.starts_with("--no-"): r2 = _ref(t, a, 5, end) if r2 >= 0 and _kind(t, r2) == 98 and not _listed(t, r2): _assign(&out, &seen, t, r2, 0, 0.0, "") continue if r < 0: return Err(_unknown(c, t, a, end)) if eq >= 0: value = a[eq + 1..] elif _kind(t, r) == 98: _assign(&out, &seen, t, r, 1, 0.0, "") continue elif i < n: value = args[i] i += 1 else: return Err(_needs(c, t, r, a, -1, end)) else: # -x, -xVALUE, -x VALUE, -abc (bools), -x=VALUE j = 1 while j < len(a): k = _short(c, a, j) if k < 0: return Err(_unknown_short(c, t, a, j)) r = _field_ref(t, c.notes[k].field) if r < 0: return Err(_no_field(c, k)) j += 1 if _kind(t, r) == 98 and (j >= len(a) or a.byte(j) != 61): _assign(&out, &seen, t, r, 1, 0.0, "") r = -1 continue if j < len(a): value = a[j..] if value.byte(0) == 61: value = value[1..] elif i < n: value = args[i] i += 1 else: return Err(_needs(c, t, r, a, k, end)) break if r < 0: continue if not _put(&out, &seen, t, r, value): return Err(_expected(_given(c, a, k, end), t, r, value)) for nt in c.notes: if nt.env != "": _env(nt, t, &out, &seen)? for field in c.required: r = _field_ref(t, field) if r < 0 or not _seen(seen, _index(t, r)): return Err(_required(field)) return Ok(rest) # The field of note nt from its environment variable, unless a flag gave it (an array # field's variable is a comma-separated list). fn _env[T](nt: Note, t: str, out: mut T, seen: mut Seen) Result[bool, str]: r = _field_ref(t, nt.field) if r < 0: return Err(_no_env_field(nt)) if _seen(seen, _index(t, r)): return Ok(false) e = os.env(nt.env, "") if e == "": return Ok(false) a = 0 while a <= len(e): z = len(e) if _listed(t, r): z = _byte_at(e, 44, a) if not _put(&out, &seen, t, r, e[a..z]): return Err(_expected("$" + nt.env, t, r, e[a..z])) a = z + 1 return Ok(true) # the index of byte b in s at or after i (len(s) if none) fn _byte_at(s: str, b: int, i: int) int: k = i while k < len(s) and s.byte(k) != b: k += 1 return k fn _required(field: str) str: return f"--{field.replace("_", "-")} is required" fn _no_env_field(nt: Note) str: return f"flag: c.env names field `{nt.field}`, which the struct doesn't have" fn _unknown_short(c: Cli, t: str, a: str, j: int) str: return _unknown(c, t, "-" + a[j], -1) # How the flag was given: a[..end], or note k's short form. fn _given(c: Cli, a: str, k: int, end: int) str: if k >= 0: return "-" + c.notes[k].short return a[..end] fn _needs(c: Cli, t: str, r: int, a: str, k: int, end: int) str: return f"{_given(c, a, k, end)} needs a value ({_what(_field(t, r))})" fn _expected(given: str, t: str, r: int, s: str) str: return f"{given}: expected {_what(_field(t, r))}, got `{s}`" fn _no_field(c: Cli, k: int) str: return f"flag: -{c.notes[k].short} names field `{c.notes[k].field}`, which the struct doesn't have" # The note whose short form is a's byte j, or -1. fn _short(c: Cli, a: str, j: int) int: b = a.byte(j) for k, n in c.notes: if len(n.short) == 1 and n.short.byte(0) == b: return k return -1 # The fields given so far: bit f of bits for field f < 63, the rest listed. struct Seen: bits: int more: [int] fn _seen(s: Seen, f: int) bool: if f < 63: return (s.bits & (1 << f)) != 0 return f in s.more # marks field f as given; whether it already was fn _mark(s: mut Seen, f: int) bool: if f < 63: b = 1 << f was = (s.bits & b) != 0 s.bits |= b return was if f in s.more: return true s.more.push(f) return false # field r = i (an int, a bool, an enum's variant), x (a float) or s (a str); an array field # starts over at its first value in this parse fn _assign[T](out: mut T, seen: mut Seen, t: str, r: int, i: int, x: float, s: str): f = _index(t, r) first = 0 if _mark(&seen, f) else 1 _set(&out, f * 2 + first, i, x, s) fn _what(f: Field) str: if f.kind == "enum": return "one of " + ", ".join(f.choices.map(fn(x) x.lower())) if f.kind == "int": return "an integer" if f.kind == "float": return "a number" if f.kind == "bool": return "true or false" return "text" # Field r = text s; false if s isn't a value of its type. fn _put[T](out: mut T, seen: mut Seen, t: str, r: int, s: str) bool: k = _kind(t, r) if k == 115: _assign(&out, &seen, t, r, 0, 0.0, s) return true if k == 105: x = s.to_int(0) if x != 0 or s.to_int(1) == 0: _assign(&out, &seen, t, r, x, 0.0, "") return true elif k == 102: x = s.trim() y = x.to_float(0.0) if x != "" and (y != 0.0 or x.to_float(1.0) == 0.0): _assign(&out, &seen, t, r, 0, y, "") return true elif k == 98: b = _truth(s) if b >= 0: _assign(&out, &seen, t, r, b, 0.0, "") return true else: # the choices are "A,B,C" after the field's letters, in variant order: compared in # place, any case a = r + 5 e = _rec_end(t, r) v = 0 while a < e: z = _byte_at(t, 44, a) if z > e: z = e if _same_fold(t, a, z, s): _assign(&out, &seen, t, r, v, 0.0, "") return true a = z + 1 v += 1 return false # 1 for true, 0 for false, -1 for neither fn _truth(s: str) int: if s == "true": return 1 if s == "false": return 0 l = s.lower() if l == "true" or l == "1" or l == "yes" or l == "on": return 1 if l == "false" or l == "0" or l == "no" or l == "off": return 0 return -1 # t[a..b] == s, ignoring ASCII case fn _same_fold(t: str, a: int, b: int, s: str) bool: if b - a != len(s): return false for k in 0..len(s): x = t.byte(a + k) y = s.byte(k) if x != y and x | 32 != y | 32: return false if x != y and (x | 32 < 97 or x | 32 > 122): return false return true # "unknown flag", for given[..end] (all of it when end < 0) fn _unknown(c: Cli, t: str, a: str, end: int) str: given = a if end < 0 else a[..end] fields = _fields(t) best = "" best_d = 3 if given.starts_with("--"): g = given[2..] for f in fields: d = _distance(g, f.long) if d < best_d: best_d = d best = "--" + f.long msg = f"unknown flag {given}" if best != "": msg += f" (did you mean {best}?)" name = c.name if name == "": name = "the program" return msg + f"; {name} --help lists the flags" # Levenshtein distance, for "did you mean" fn _distance(a: str, b: str) int: prev: [int] = [] for j in 0..len(b) + 1: prev.push(j) for i in 1..len(a) + 1: cur: [int] = [i] for j in 1..len(b) + 1: cost = 1 if a.byte(i - 1) == b.byte(j - 1): cost = 0 cur.push(min(min(prev[j] + 1, cur[j - 1] + 1), prev[j - 1] + cost)) prev = cur return prev[len(b)] ``` ## library/std/flag/usage.tn ```tern # flag: the --help text, from the struct's fields and the Cli's notes. import json fn usage[T](c: Cli, out: T) str: fields = _fields(_table(out).unwrap_or("")) defaults = json.encode(out) name = c.name if name == "": name = "program" u = "" if c.about != "": u += f"{name}: {c.about}\n\n" u += f"usage: {name} [flags]" if c.positional != "": u += " " + c.positional u += "\n\nflags:\n" left: [str] = [] right: [str] = [] for f in fields: nt = Note(f.name, "", "", "") for x in c.notes: if x.field == f.name: nt = x l = " " if nt.short != "": l = f" -{nt.short}, " l += "--" + f.long if f.kind == "enum": l += " " + "|".join(f.choices.map(fn(x) x.lower())) elif f.kind != "bool": l += " " + f.kind if f.list: l += "..." left.push(l) r = nt.help d = _default(defaults, f.name) # its JSON text shown = "" if f.list or f.optional or d == "": shown = "" elif f.kind == "bool": if d == "true": shown = "true" elif f.kind == "str": if d != "\"\"": shown = d elif f.kind == "enum": shown = d[1..len(d) - 1].lower() else: shown = d if shown != "": if r != "": r += " " r += f"(default {shown})" if nt.env != "": if r != "": r += " " r += f"[${nt.env}]" if f.list: if r != "": r += " " r += "(repeatable)" right.push(r) left.push(" -h, --help") right.push("show this help") w = 0 for l in left: w = max(w, len(l)) for k in 0..len(left): line = left[k] if right[k] != "": line += " ".repeat(w - len(left[k]) + 2) + right[k] u += line + "\n" return u # The JSON text of field `name` in enc, an encoded struct ({"a":1,"b":"x"}); "" if missing. # A key inside a string can't match: its quotes are escaped there. fn _default(enc: str, name: str) str: i = enc.find("{\"" + name + "\":") if i < 0: i = enc.find(",\"" + name + "\":") if i < 0: return "" a = i + len(name) + 4 if a >= len(enc): return "" if enc.byte(a) == 34: b = a + 1 while b < len(enc) and enc.byte(b) != 34: if enc.byte(b) == 92: b += 1 b += 1 return enc[a..b + 1] b = a while b < len(enc) and enc.byte(b) != 44 and enc.byte(b) != 125: b += 1 return enc[a..b] ``` ## library/std/fmt.tn ```tern # fmt: formatting numbers and padding text (f-strings have no format specs; use these). # fmt.fixed(x: float, digits: int) str fixed(3.14159, 2) == "3.14", fixed(2.0, 3) == "2.000" # fmt.pad_left(s: str, width: int, fill: str) str pad_left("7", 3, "0") == "007" # fmt.pad_right(s: str, width: int, fill: str) str pad_right("ab", 4, ".") == "ab.." # fmt.zero_pad(n: int, width: int) str zero_pad(5, 2) == "05", zero_pad(-5, 3) == "-05" # fmt.thousands(n: int) str thousands(1234567) == "1,234,567" # fmt.cents(n: int) str cents(1205) == "12.05", cents(-5) == "-0.05" # `fill` is one character; text already `width` bytes or longer is returned unchanged. import math fn pad_left(s: str, width: int, fill: str) str: if len(s) >= width: return s return fill.repeat(width - len(s)) + s fn pad_right(s: str, width: int, fill: str) str: if len(s) >= width: return s return s + fill.repeat(width - len(s)) fn zero_pad(n: int, width: int) str: if n < 0: return "-" + pad_left(str(-n), width - 1, "0") return pad_left(str(n), width, "0") fn thousands(n: int) str: digits = str(abs(n)) out = "" for i in 0..len(digits): if i > 0 and (len(digits) - i) % 3 == 0: out += "," out += digits[i] if n < 0: return "-" + out return out fn _scaled(n: int, digits: int) str: # n is x * 10^digits, rounded; put the decimal point back in sign = "" v = n if v < 0: sign = "-" v = -v if digits <= 0: return sign + str(v) unit = 1 for i in 0..digits: unit *= 10 return sign + str(v / unit) + "." + pad_left(str(v % unit), digits, "0") fn fixed(x: float, digits: int) str: scale = 1.0 for i in 0..digits: scale *= 10.0 return _scaled(int(math.round(x * scale)), digits) fn cents(n: int) str: return _scaled(n, 2) ``` ## library/std/fs.tn ```tern # fs: files and directories. # fs.read(path) str ("" if it can't be read; fs.error() says why) # fs.write(path, text) bool fs.append(path, text) bool # fs.exists(path) bool fs.remove(path) bool fs.rename(from, to) bool # fs.list(dir) [str] fs.mkdir(path) bool # fs.list_marked(dir) [str] fs.list with "/" after each directory's name (no stat per name) # fs.stat(path) Info Info fields: exists, is_dir, size (bytes), modified (seconds since # 1970, like time.now()); exists is false if it can't be read # fs.walk(dir) [str] every file below dir (not the directories), as dir/sub/name, sorted # fs.copy(from, to) bool a file's bytes and permissions # fs.chmod(path, mode) bool set permission bits: fs.chmod(p, 0o600) (only the owner can read or # write it: for keys and secrets); a no-op on WebAssembly # fs.remove_all(path) bool a file, or a directory with everything in it (true if it's gone) # fs.realpath(path) str the absolute path with symlinks, "." and ".." resolved ("" if it # doesn't exist) # fs.temp_dir(prefix) str a new empty directory in $TMPDIR (or /tmp), e.g. /tmp/prefixA1b2C3 # fs.error() str why the last fs call on this thread failed, e.g. # "can't read notes.txt: No such file or directory" ("" after a success) # Reading or writing a piece at a time (big files, logs): # f = fs.open(path) File for reading f = fs.create(path) File for writing (emptied) # f = fs.open_append(path) File check f.ok(); fs.error() says why not # f.read_line() str the next line with its "\n" ("" at the end of the file) # f.read_some(max) str up to max bytes: what has arrived, on a pipe ("" at the end) # f.read_at(off, n) str n bytes at offset off, fewer at the end ("" past it); doesn't move # where read_line / read_some read, and tasks may share the file # f.put(text) bool write text (buffered) f.flush() bool write out what was put so far # f.close() bool (the program's end writes out and closes the rest) # # f = fs.open("big.log") # line = f.read_line() # while line != "": # handle(line.trim()) # line = f.read_line() # f.close() # # In a task, file calls run on the task's thread (a big read holds up that core's other # tasks briefly; for very large files use task.blocking). struct Info: exists: bool is_dir: bool size: int modified: float struct File: h: int fn open(path: str) File: return File(_open(path, 0)) fn create(path: str) File: return File(_open(path, 1)) fn open_append(path: str) File: return File(_open(path, 2)) fn ok(f: File) bool: return f.h >= 0 fn read_line(f: File) str: return _read_line(f.h) fn read_some(f: File, max: int) str: return _read(f.h, max) fn read_at(f: File, off: int, n: int) str: return _read_at(f.h, off, n) fn put(f: File, text: str) bool: return _write(f.h, text) fn flush(f: File) bool: return _flush(f.h) fn close(f: File) bool: return _close(f.h) fn walk(dir: str) [str]: out: [str] = [] _walk(dir, &out) return out.sort_by(fn(x) x) fn _walk(dir: str, out: mut [str]): for n in list(dir): p = dir + "/" + n if stat(p).is_dir: _walk(p, &out) else: out.push(p) # ---------------------------------------------------------------- with Result # The same operations returning Result, the error saying what failed and why # ("fs: can't read notes.txt: No such file or directory"): # text = fs.try_read(path)? n = fs.try_write(path, text)? names = fs.try_list(dir)? # info = fs.try_stat(path)? f = fs.try_open(path)? fn try_read(path: str) Result[str, str]: s = read(path) if error() != "": return Err("fs: " + error()) return Ok(s) fn try_write(path: str, text: str) Result[int, str]: if not write(path, text): return Err("fs: " + error()) return Ok(len(text)) fn try_append(path: str, text: str) Result[int, str]: if not append(path, text): return Err("fs: " + error()) return Ok(len(text)) fn try_list(dir: str) Result[[str], str]: names = list(dir) if error() != "": return Err("fs: " + error()) return Ok(names) fn try_stat(path: str) Result[Info, str]: i = stat(path) if not i.exists: return Err("fs: " + error()) return Ok(i) fn try_open(path: str) Result[File, str]: f = open(path) if not f.ok(): return Err("fs: " + error()) return Ok(f) ``` ## library/std/heap.tn ```tern # heap: a heap of ints, floats or strs: smallest first (min_heap) or largest first # (max_heap). A heap is a value like any other; push and pop take it `mut`, so h.push(x) is # push(&h, x). For values that aren't ordered themselves (structs), use pqueue. # # import heap # h = heap.min_heap[int]() # for x in [5, 1, 3]: # h.push(x) # while h.pop() is Some(x): ... # 1, 3, 5 # # heap.min_heap[T]() Heap[T] heap.max_heap[T]() Heap[T] (T: int, float or str) # heap.heap_of(xs: [T], max: bool) Heap[T] from an array, in O(n) # h.push(x) h.pop() Option[T] h.peek() Option[T] h.len() h.is_empty() # h.drain() [T] every value, in pop order, leaving h empty # # A 4-ary heap; sifting moves a hole and writes each element once, instead of swapping pairs. struct Heap[T]: items: [T] max: bool big: bool # has held _BIG values: when it empties, its array is let go fn min_heap[T: Ord]() Heap[T]: return Heap[T]([], false, false) fn max_heap[T: Ord]() Heap[T]: return Heap[T]([], true, false) fn heap_of[T: Ord](xs: [T], max: bool) Heap[T]: h = Heap[T](xs, max, len(xs) >= _BIG) i = (len(xs) - 2) >> 2 while i >= 0: if max: _down_max(&h.items, i) else: _down_min(&h.items, i) i -= 1 return h fn push[T: Ord](h: mut Heap[T], x: T): h.items.push(x) if len(h.items) == _BIG: h.big = true if h.max: _up_max(&h.items, x) else: _up_min(&h.items, x) # A heap that has grown big gives its memory back when pop empties it (an array keeps its # capacity otherwise); a small one keeps it, so a heap that keeps emptying and refilling # doesn't allocate each time. const _BIG = 4096 fn pop[T: Ord](h: mut Heap[T]) Option[T]: n = len(h.items) if n == 0: return None top = h.items[0] if n == 1 and h.big: h.items = [] h.big = false return Some(top) last = h.items.pop() if n > 1: if h.max: _refill_max(&h.items, last) else: _refill_min(&h.items, last) return Some(top) # The sifts work on the array itself, one loop for each order: the order is decided once # per call, not at every comparison. The heap is 4-ary (children of i at 4i+1 .. 4i+4): half # the levels of a binary heap, and a node's children sit side by side in memory, so a pop # that walks down to a leaf takes about half the cache misses. # # After a pop the hole at the root moves down to a leaf, always taking the best child, then # x (the old last element, usually among the worst) rises from there: fewer comparisons than # sifting x down from the root (Rust's BinaryHeap does the same on its binary heap). fn _up_min[T: Ord](a: mut [T], x: T): i = len(a) - 1 while i > 0: p = (i - 1) >> 2 if x >= a[p]: break a[i] = a[p] i = p a[i] = x fn _down_min[T: Ord](a: mut [T], start: int): n = len(a) i = start x = a[i] while true: c = 4 * i + 1 if c >= n: break m = c k = c + 1 e = c + 4 if e > n: e = n while k < e: if a[k] < a[m]: m = k k += 1 if a[m] >= x: break a[i] = a[m] i = m a[i] = x fn _refill_min[T: Ord](a: mut [T], x: T): n = len(a) i = 0 c = 1 while c + 3 < n: m = c if a[c + 1] < a[c]: m = c + 1 m2 = c + 2 if a[c + 3] < a[c + 2]: m2 = c + 3 if a[m2] < a[m]: m = m2 a[i] = a[m] i = m c = 4 * i + 1 if c < n: m = c k = c + 1 while k < n: if a[k] < a[m]: m = k k += 1 a[i] = a[m] i = m while i > 0: p = (i - 1) >> 2 if x >= a[p]: break a[i] = a[p] i = p a[i] = x fn _up_max[T: Ord](a: mut [T], x: T): i = len(a) - 1 while i > 0: p = (i - 1) >> 2 if x <= a[p]: break a[i] = a[p] i = p a[i] = x fn _down_max[T: Ord](a: mut [T], start: int): n = len(a) i = start x = a[i] while true: c = 4 * i + 1 if c >= n: break m = c k = c + 1 e = c + 4 if e > n: e = n while k < e: if a[k] > a[m]: m = k k += 1 if a[m] <= x: break a[i] = a[m] i = m a[i] = x fn _refill_max[T: Ord](a: mut [T], x: T): n = len(a) i = 0 c = 1 while c + 3 < n: m = c if a[c + 1] > a[c]: m = c + 1 m2 = c + 2 if a[c + 3] > a[c + 2]: m2 = c + 3 if a[m2] > a[m]: m = m2 a[i] = a[m] i = m c = 4 * i + 1 if c < n: m = c k = c + 1 while k < n: if a[k] > a[m]: m = k k += 1 a[i] = a[m] i = m while i > 0: p = (i - 1) >> 2 if x <= a[p]: break a[i] = a[p] i = p a[i] = x fn peek[T](h: Heap[T]) Option[T]: if len(h.items) == 0: return None return Some(h.items[0]) fn len[T](h: Heap[T]) int: return len(h.items) fn is_empty[T](h: Heap[T]) bool: return len(h.items) == 0 fn drain[T: Ord](h: mut Heap[T]) [T]: out: [T] = [] while pop(&h) is Some(x): out.push(x) return out ``` ## library/std/html/data.tn ```tern # html: the render's data, a JSON value, parsed into flat arrays: values in _Data.vals (a # container's members are a run of _Data.mem), strings left in the text unless they have # escapes (those are decoded into _Data.dec). import simd const _NULL = 0 const _FALSE = 1 const _TRUE = 2 const _NUM = 3 const _STR = 4 const _ARR = 5 const _OBJ = 6 struct _V: kind: int s: int # a string's or number's text: src[s..s + n], or dec[-1 - s..] when s < 0; n: int # an array's or object's JSON text: src[s..s + n] first: int # an array's or object's members: mem[first..first + count]; anything count: int # else's JSON text: src[first..first + count] (for {{json}}) # a value's JSON text: src[_raw(v).._raw(v) + _rawn(v)] fn _raw(v: _V) int: return v.s if v.kind >= _ARR else v.first fn _rawn(v: _V) int: return v.n if v.kind >= _ARR else v.count struct _M: ks: int # an object member's key (as _V.s: in src, or decoded in dec) kn: int v: int # the value: an index into vals struct _Data: src: str dec: str vals: [_V] mem: [_M] stk: [_M] # members of the containers being parsed i: int depth: int fn _parse_data(src: str) _Data: d = _Data(src, "", [], [], [], 0, 0) return d fn _ws(d: mut _Data): n = len(d.src) while d.i < n: c = d.src.byte(d.i) if c != 32 and c != 10 and c != 13 and c != 9: return d.i += 1 fn _hexval(c: int) int: if c >= 48 and c <= 57: return c - 48 if c >= 97 and c <= 102: return c - 87 if c >= 65 and c <= 70: return c - 55 return -1 # four hex digits at q (fewer if they'd reach e; a non-hex digit counts as 0) fn _hex4(s: str, q: int, e: int) int: cp = 0 for k in 1..5: if q + k >= e: break v = _hexval(s.byte(q + k)) cp = cp * 16 + (0 if v < 0 else v) return cp fn _utf8(out: mut str, cp: int): if cp < 0x80: out += chr(cp) elif cp < 0x800: out += chr(0xC0 | cp >> 6) + chr(0x80 | (cp & 63)) elif cp < 0x10000: out += chr(0xE0 | cp >> 12) + chr(0x80 | (cp >> 6 & 63)) + chr(0x80 | (cp & 63)) else: out += chr(0xF0 | cp >> 18) + chr(0x80 | (cp >> 12 & 63)) + chr(0x80 | (cp >> 6 & 63)) + chr(0x80 | (cp & 63)) # a string at d.i (its opening quote): its text as (s, n), see _V.s fn _str(d: mut _Data, s: mut int, n: mut int) bool: src = d.src e = len(src) start = d.i + 1 i = start plain = true quote = simd.splat(34) bslash = simd.splat(92) while i < e: if i + 16 <= e: # 16 bytes without " or \: skip them v = simd.load(src, i) k = simd.first_lane(simd.eq(v, quote) | simd.eq(v, bslash)) i += k if k == 16: continue c = src.byte(i) if c == 34: break if c == 92: plain = false i += 1 i += 1 if i >= e: return false d.i = i + 1 if plain: s = start n = i - start return true base = len(d.dec) q = start run = start while q < i: if src.byte(q) != 92: q += 1 continue d.dec += src[run..q] q += 1 c = src.byte(q) if c == 110: d.dec += "\n" elif c == 116: d.dec += "\t" elif c == 114: d.dec += "\r" elif c == 98: d.dec += "\x08" elif c == 102: d.dec += "\x0c" elif c == 117: cp = _hex4(src, q, i) q += 4 if cp >= 0xD800 and cp < 0xDC00 and q + 6 < i and src.byte(q + 1) == 92 and src.byte(q + 2) == 117: lo = _hex4(src, q + 2, i) # a surrogate pair if lo >= 0xDC00 and lo < 0xE000: cp = 0x10000 + ((cp - 0xD800) << 10) + (lo - 0xDC00) q += 6 _utf8(&d.dec, cp) else: d.dec += chr(c) q += 1 run = q if run < i: d.dec += src[run..i] s = -1 - base n = len(d.dec) - base return true fn _lit(s: str, i: int, w: str) bool: if len(s) - i < len(w): return false for k in 0..len(w): if s.byte(i + k) != w.byte(k): return false return true # a number's bytes (as the C took them: also NUL) fn _numch(c: int) bool: return (c >= 48 and c <= 57) or c == 43 or c == 45 or c == 46 or c == 101 or c == 69 or c == 0 # the value at d.i: its index in d.vals, or -1 if it isn't JSON. A value is pushed after its # members (so the root is the last one). fn _value(d: mut _Data) int: _ws(&d) n = len(d.src) if d.i >= n: return -1 d.depth += 1 if d.depth > 200: return -1 raw = d.i c = d.src.byte(raw) kind = _NULL s = 0 sn = 0 first = 0 count = 0 if c == 34: if not _str(&d, &s, &sn): return -1 kind = _STR elif c == 91 or c == 123: obj = c == 123 close = 125 if obj else 93 d.i += 1 base = len(d.stk) _ws(&d) if d.i < n and d.src.byte(d.i) == close: d.i += 1 else: while true: ks = 0 kn = 0 _ws(&d) if obj: if d.i >= n or d.src.byte(d.i) != 34 or not _str(&d, &ks, &kn): return -1 _ws(&d) if d.i >= n or d.src.byte(d.i) != 58: return -1 d.i += 1 kid = _value(&d) if kid < 0: return -1 d.stk.push(_M(ks, kn, kid)) _ws(&d) if d.i < n and d.src.byte(d.i) == 44: d.i += 1 continue if d.i < n and d.src.byte(d.i) == close: d.i += 1 break return -1 first = len(d.mem) count = len(d.stk) - base for k in base..len(d.stk): d.mem.push(d.stk[k]) while len(d.stk) > base: d.stk.pop() kind = _OBJ if obj else _ARR s = raw sn = d.i - raw elif _lit(d.src, raw, "true"): kind = _TRUE d.i += 4 elif _lit(d.src, raw, "false"): kind = _FALSE d.i += 5 elif _lit(d.src, raw, "null"): d.i += 4 else: while d.i < n and _numch(d.src.byte(d.i)): d.i += 1 if d.i == raw: return -1 kind = _NUM s = raw sn = d.i - raw if kind < _ARR: first = raw count = d.i - raw d.depth -= 1 d.vals.push(_V(kind, s, sn, first, count)) return len(d.vals) - 1 # is the number text src[a..a + n] nonzero, as C's strtod reads it (the longest prefix that is # a number; past a NUL nothing counts)? strtod is exact, so only an exponent below about -324 # makes nonzero digits 0: the digits are shortened (with a sticky digit) to fit to_float. fn _num_nonzero(src: str, a: int, n: int) bool: e = a + n i = a if i < e and (src.byte(i) == 43 or src.byte(i) == 45): i += 1 isig = i digits = "" lead = -1 # the position (in digits) of the first nonzero digit intn = 0 # digits before the point seen = false while i < e and src.byte(i) >= 48 and src.byte(i) <= 57: if lead < 0 and src.byte(i) != 48: lead = len(digits) digits += src[i] intn += 1 seen = true i += 1 if i < e and src.byte(i) == 46: i += 1 while i < e and src.byte(i) >= 48 and src.byte(i) <= 57: if lead < 0 and src.byte(i) != 48: lead = len(digits) digits += src[i] seen = true i += 1 if isig == e: # a sign alone (only a root value: see below) return _inf_nan(src, e) if e - isig == 1 and src.byte(isig) == 48 and e < len(src) and (src.byte(e) == 120 or src.byte(e) == 88): return _hex_nonzero(src, e + 1) if not seen or lead < 0: return false exp = 0 if i < e and (src.byte(i) == 101 or src.byte(i) == 69): j = i + 1 neg = false if j < e and (src.byte(j) == 43 or src.byte(j) == 45): neg = src.byte(j) == 45 j += 1 if j < e and src.byte(j) >= 48 and src.byte(j) <= 57: while j < e and src.byte(j) >= 48 and src.byte(j) <= 57: if exp < 100000: exp = exp * 10 + src.byte(j) - 48 j += 1 if neg: exp = -exp lexp = intn - 1 - lead + exp # the first nonzero digit is that power of 10 if lexp > -320: return true if lexp < -330: return false sig = digits[lead..] if len(sig) > 40: sticky = false for k in 40..len(sig): if sig.byte(k) != 48: sticky = true sig = sig[..40] + ("1" if sticky else "") return ("0." + sig + "e" + str(lexp + 1)).to_float(0.0) != 0.0 # strtod reads on past a number's bytes when they are a sign alone or [sign] 0 and an x # follows: inf, nan or a hex float. That happens only for the root value, which may be followed # by anything (inside an array or object a number is followed by , ] } or space). fn _inf_nan(s: str, i: int) bool: w = s[i..min(i + 3, len(s))].lower() return w == "inf" or w == "nan" fn _hexd(c: int) int: return _hexval(c) # the hex float after "0x" at i is nonzero (0x alone is the number 0) fn _hex_nonzero(s: str, i0: int) bool: e = len(s) i = i0 lead = -1 # the first nonzero digit's position among the digits, and its value leadv = 0 nd = 0 intn = 0 more = false # a nonzero bit after the leading one while i < e and _hexd(s.byte(i)) >= 0: v = _hexd(s.byte(i)) if lead < 0 and v != 0: lead = nd leadv = v elif lead >= 0 and v != 0: more = true nd += 1 intn += 1 i += 1 if i < e and s.byte(i) == 46: i += 1 while i < e and _hexd(s.byte(i)) >= 0: v = _hexd(s.byte(i)) if lead < 0 and v != 0: lead = nd leadv = v elif lead >= 0 and v != 0: more = true nd += 1 i += 1 if nd == 0 or lead < 0: return false exp = 0 if i < e and (s.byte(i) == 112 or s.byte(i) == 80): j = i + 1 neg = false if j < e and (s.byte(j) == 43 or s.byte(j) == 45): neg = s.byte(j) == 45 j += 1 if j < e and s.byte(j) >= 48 and s.byte(j) <= 57: while j < e and s.byte(j) >= 48 and s.byte(j) <= 57: if exp < 100000: exp = exp * 10 + s.byte(j) - 48 j += 1 if neg: exp = -exp top = 3 while (leadv >> top) & 1 == 0: top -= 1 if leadv & ((1 << top) - 1) != 0: more = true bexp = 4 * (intn - 1 - lead) + top + exp # the leading bit is 2 to this power if bexp >= -1074: return true return bexp == -1075 and more # above half the smallest double rounds up ``` ## library/std/html/escape.tn ```tern # html: escaping, 16 bytes at a time. A block's bytes to replace as a mask, then from one to # the next (the clean run before each is one append, its entity another). import simd fn escape(s: str) str: i = _first_esc(s, 0, len(s)) if i == len(s): return s out = "" out.reserve(len(s) + _growth(s, i)) # exactly its length _esc_into(&out, s, 0, len(s)) return out # how many bytes escaping s[i..] adds: & and ' 4 each, < and > 3, " 5 fn _growth(s: str, i0: int) int: n = len(s) g = 0 i = i0 while i < n: v = simd.load_upto(s, i) w = (simd.eq(v | simd.splat(2), simd.splat(62)) & simd.splat(3)) | (simd.eq(v | simd.splat(1), simd.splat(39)) & simd.splat(4)) | (simd.eq(v, simd.splat(34)) & simd.splat(5)) g += simd.sum(w) # (load_upto's zeros past the end add 0) i += 16 return g # lanes 0 to 15, to compare with a count: simd.lt(_lanes(), simd.splat(k)) is the first k fn _lanes() bytes16: return simd.of("\x00\x01\x02\x03\x04\x05\x06\x07\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f") # the lanes of the 16 bytes at i that are before b fn _upto(i: int, b: int) bytes16: return simd.lt(_lanes(), simd.splat(min(b - i, 16))) # & < > " ' : 255 in their lanes (< and > are 60 and 62, & and ' are 38 and 39) fn _html16(v: bytes16) bytes16: return simd.eq(v | simd.splat(2), simd.splat(62)) | simd.eq(v | simd.splat(1), simd.splat(39)) | simd.eq(v, simd.splat(34)) # < > & : the bytes {{json}} writes as \u escapes fn _json16(v: bytes16) bytes16: return simd.eq(v | simd.splat(2), simd.splat(62)) | simd.eq(v, simd.splat(38)) # the first byte of s[i..b] that escape replaces (b if none) fn _first_esc(s: str, i0: int, b: int) int: i = i0 while i < b: k = simd.first_lane(_html16(simd.load_upto(s, i)) & _upto(i, b)) if k < 16: return i + k i += 16 return b fn _entity(c: int) str: if c == 38: return "&" if c == 60: return "<" if c == 62: return ">" if c == 34: return """ return "'" # out += s[a..b] escaped fn _esc_into(out: mut str, s: str, a: int, b: int): run = a i = a while i < b: m = _html16(simd.load_upto(s, i)) & _upto(i, b) if simd.any(m): bits = simd.bits(m) while bits != 0: k = i + simd.first(bits) out += s[run..k] + _entity(s.byte(k)) run = k + 1 bits &= bits - 1 i += 16 out += s[run..b] # out += s[a..b] with < > & as < > & (safe inside " init = "window.ui = SwaggerUIBundle({ url: '/openapi.json', dom_id: '#swagger-ui', deepLinking: true, persistAuthorization: true });" r.get("/openapi.json", fn(req) roost.json(doc)) r.get("/docs", fn(req) roost.html(page)) fn js(req: roost.Request) roost.Response: res = roost.text(init) res.content_type = "text/javascript; charset=utf-8" return res r.get("/docs/init.js", js) fn html_text(s: str) str: return s.replace("&", "&").replace("<", "<").replace(">", ">").replace("'", "'") # ---------------------------------------------------------------- TypeScript # typescript(spec): a type for every schema, and createClient() with one typed method per # operation (named by its operationId): path parameters, then the body, then the query. # int64 becomes number: exact up to 2^53. fn _ts_name(def: str, taken: {str: int}) str: short = def d = def.find(".") while d >= 0: short = short[d + 1..] d = short.find(".") if taken.get(short, 0) > 1: # store.Stats and forward.Stats: StoreStats, ForwardStats out = "" for part in def.split("."): if len(part) > 0: out += part[0].upper() + part[1..] return out return short fn _ts_type(v: json.Value, names: {str: str}) str: ref = v.field("$ref").str_or("") if ref != "": n = ref[ref.find("/schemas/") + 9..] if ref.starts_with("#/$defs/"): n = ref[8..] return names.get(n, "unknown") alts = v.field("anyOf").elems() # Option[T]: T or null if len(alts) == 0: alts = v.field("oneOf").elems() # an enum with payloads if len(alts) > 0: return " | ".join(alts.map(fn(a) _ts_type(a, names))) if v.field("const") is Str(c): # one variant's name return json_quote(c) labels = v.field("enum").elems() # a string enum: "Active" | "Banned" if len(labels) > 0: return " | ".join(labels.map(fn(l) json_quote(l.str_or("")))) t = v.field("type").str_or("") if t == "null": return "null" if t == "integer" or t == "number": return "number" if t == "string": return "string" if t == "boolean": return "boolean" if t == "array": it = _ts_type(v.field("items"), names) if it.contains(" ") or it.contains("<"): return "Array<" + it + ">" return it + "[]" if t == "object": ap = v.field("additionalProperties") if ap is Obj(_, _): # (false: a closed object, not a map) return "Record" props = v.field("properties") parts: [str] = [] for k in props.fields(): parts.push(_ts_key(k) + ": " + _ts_type(props.field(k), names)) return "{ " + "; ".join(parts) + " }" return "unknown" fn _ts_key(k: str) str: for i in 0..len(k): c = k.byte(i) if not ((c >= 97 and c <= 122) or (c >= 65 and c <= 90) or (c >= 48 and c <= 57 and i > 0) or c == 95 or c == 36): return json_quote(k) return k fn _ts_schema(schema: str, names: {str: str}) str: match json.parse(schema): Ok(v): return _ts_type(v, names) Err(_): return "unknown" fn _ts_param_type(ty: str) str: if ty == "integer" or ty == "number": return "number" if ty == "boolean": return "boolean" return "string" fn typescript(s: Spec) str: defs: {str: str} = {} for o in s.ops: _split(o.request, &defs) _split(o.response, &defs) names = _short_names(defs) out = "// Generated from the " + s.title + " API (" + s.version + ") by Tern's openapi.typescript: don't edit.\n\n" for k, v in defs: out += "export type " + names[k] + " = " + _ts_schema(v, names) + ";\n" out += "\nexport class ApiError extends Error {\n constructor(public status: number, public body: string) {\n super(`${status}: ${body}`);\n }\n}\n\n" out += "export type ClientOptions = { baseUrl?: string; token?: string; fetch?: typeof fetch; init?: RequestInit };\n\n" out += "export function createClient(opts: ClientOptions = {}) {\n" out += " const base = opts.baseUrl ?? \"\";\n" out += " async function call(method: string, path: string, query: object | undefined, body: unknown, kind: \"json\" | \"text\" | \"none\"): Promise {\n" out += " let url = base + path;\n" out += " if (query) {\n const u = new URLSearchParams();\n for (const [k, v] of Object.entries(query)) if (v !== undefined && v !== null && v !== \"\") u.set(k, String(v));\n const q = u.toString();\n if (q) url += \"?\" + q;\n }\n" out += " const headers: Record = { ...(opts.init?.headers as Record | undefined) };\n" out += " if (opts.token) headers.Authorization = `Bearer ${opts.token}`;\n" out += " if (body !== undefined) headers[\"Content-Type\"] = \"application/json\";\n" out += " const res = await (opts.fetch ?? fetch)(url, { credentials: \"same-origin\", ...opts.init, method, headers, body: body === undefined ? undefined : JSON.stringify(body) });\n" out += " if (!res.ok) throw new ApiError(res.status, await res.text());\n" out += " if (kind === \"none\") return undefined as T;\n" out += " return (kind === \"text\" ? await res.text() : await res.json()) as T;\n }\n" out += " return {\n" for o in s.ops: args: [str] = [] pp: [str] = [] qp: [str] = [] for p in o.params: if p.loc == "path": pp.push(_ts_key(p.name) + ": string") else: qp.push(_ts_key(p.name) + "?: " + _ts_param_type(p.ty)) if len(pp) > 0: args.push("p: { " + "; ".join(pp) + " }") body = "undefined" if o.request != "": args.push("body: " + _ts_schema(_split(o.request, &defs), names)) body = "body" query = "undefined" if len(qp) > 0: args.push("q: { " + "; ".join(qp) + " } = {}") query = "q" ret = "void" kind = "none" if o.response != "": ret = _ts_schema(_split(o.response, &defs), names) kind = "json" elif o.text: ret = "string" kind = "text" path = _openapi_path(o.path) for p in o.params: if p.loc == "path": path = path.replace("{" + p.name + "}", "${encodeURIComponent(p." + p.name + ")}") out += " /** " + o.summary.replace("*/", "* /") + " */\n" out += " " + o.op_id + ": (" + ", ".join(args) + ") => call<" + ret + ">(" + json_quote(o.method) + ", `" + path + "`, " + query + ", " + body + ", " + json_quote(kind) + "),\n" out += " };\n}\n" return out ``` ## deps/pg/pg.tn ```tern # pg: a PostgreSQL client for tasks: a connection pool, parameters bound safely, rows read # into your structs, and transactions that belong to the task that started them. # # import pg # import json # # struct World: # id: int # randomNumber: int # # db = pg.pool("postgres://user:password@localhost:5432/app", 16) # 16 connections, shared by every task # if not db.ok(): # print(pg.error()) # why the first connection failed # n = db.exec("UPDATE world SET randomNumber = $1 WHERE id = $2", ["42", "7"]) # rows changed (-1: error) # rows = db.query("SELECT id, name FROM users WHERE id = $1", ["7"]) # [{str: str}] # worlds: [World] = [] # json.decode(db.query_json("SELECT id, randomNumber FROM world LIMIT 5", []), &worlds) # w = World(0, 0) # json.decode(db.query_row_json("SELECT * FROM world WHERE id = $1", ["7"]), &w) # # db.begin() # from here until commit/rollback, this task's # db.exec("INSERT INTO log (msg) VALUES ($1)", ["hi"]) # statements run on one connection, # db.commit() # which no other task can use meanwhile # # Functions (a Pool is shared by copying it, like a channel): # pg.pool(url str, size int) Pool at most size connections, opened as needed (the first # one right away, so p.ok() can report a bad url or password) # pg.connect(url str) Pool pg.pool(url, 1) # p.ok() bool pg.error() str: why the last call failed ("" after a success) # p.exec(sql str, params [str]) int rows affected, or -1 (pg.error() has the server's message, # e.g. "ERROR 23505: duplicate key value violates unique constraint ...") # p.query(sql str, params [str]) [{str: str}] every value as text; NULL is "" # p.query_json(sql str, params [str]) str the rows as a JSON array: numbers, booleans and # null stay typed, so json.decode fills structs # p.query_row_json(sql str, params [str]) str the first row as a JSON object ("null" if none) # p.begin() bool p.commit() bool p.rollback() bool # p.run_script(sql str) Result[int, str] several statements (a schema, a migration): the # simple query protocol, no parameters # p.describe(sql str) Result[Statement, str] the types of its parameters and columns, # without running it (see "describing a statement" below) # pg.array_int(xs) str array_float array_bool array_str a list as one parameter: # `WHERE id = ANY($1)` with [pg.array_int(ids)] # pg.NULL a parameter that is NULL # p.stats() PoolStats size open idle in_use waiting waits wait_ms timeouts opened closed # p.prune() closes the connections idle longer than max_idle (the # pool also does this by itself now and then) # p.close() closes the idle connections # # Parameters are $1, $2, ... and are sent separately from the SQL (no injection), as text; # PostgreSQL converts them to the column types; pg.NULL is NULL. Statements are prepared on # each connection the first time and reused after that. # # Transactions are bound to the task: any statement that leaves its connection inside a # transaction (begin(), or exec("BEGIN")) keeps that connection for the task until the # transaction ends, so other tasks never see or join it. A task that waits (a channel, a # socket, a sleep) in the middle of a transaction keeps holding its connection. # # URL: postgres://user:password@host:port/dbname?sslmode=require (sslmode=require or # verify-full need `import tls`; the default is no TLS); &application_name=app names the # connections in pg_stat_activity (default: tern). Login: SCRAM-SHA-256 (PostgreSQL's # default) or password. Not available on wasm. # The pool: &pool_timeout=30s (how long a statement waits for a free connection, then fails), # &max_lifetime=1h (a connection is replaced after this long), &max_idle=30m (an idle one is # closed), &statement_cache=off (no prepared statements kept: PgBouncer); 0 turns a limit off. # A connection the server closed is noticed and replaced before it is used. # # The protocol is Tern too: wire.tn. import crypto import json import net import sql import task import time struct Pool: url: str size: int free: chan[Conn] # idle connections; one not live is a slot not connected yet state: shared[_State] shards: [shared[_Shard]] # transactions' connections and the waiting counts, by task (see _Shard) key: str # the task-local that marks a task holding one of its connections first_ok: bool timeout_ms: int # ?pool_timeout: how long a task waits for a free connection (0: forever) lifetime_ms: int # ?max_lifetime: a connection is replaced after this long (0: never) idle_ms: int # ?max_idle: an idle connection is closed after this long (0: never) # what p.stats() reports struct PoolStats: size: int # the most connections it opens open: int # connections open now idle: int # ... of them not in use in_use: int # ... running a statement, or held by a task's transaction waiting: int # tasks waiting for a connection now waits: int # times a task had to wait (every connection was in use) wait_ms: int # ... and the milliseconds it waited, added up timeouts: int # waits that gave up after pool_timeout opened: int # connections opened, ever closed: int # ... and closed (broken, too old, idle too long, p.close()) # What the tasks using a pool share, behind a lock. A statement that finds a free connection # doesn't take it: only opening and closing connections do. struct _State: stats: PoolStats # (idle and in_use are worked out by stats(); the waiting counts # are the shards') pruned: int # when idle connections were last closed (time.mono_ms) # What transactions and waits touch, split by task.id() over _SHARDS locks. Each statement in # a transaction takes its task's connection out and puts it back; with one lock for the pool, # tasks queued on it, and a lock handed to a task that has to be woken first is held all the # while it waits to run (a convoy: every statement of every transaction waited in it). struct _Shard: bound: {int: Conn} # connections held by a task's open transaction, by task.id() waiting: int # (PoolStats' fields of the same names, for these tasks) waits: int wait_ms: int timeouts: int const _SHARDS = 32 fn _shard(p: Pool, tid: int) shared[_Shard]: return p.shards[tid % len(p.shards)] fn pool(url: str, size: int) Pool: n = max(size, 1) free: chan[Conn] = task.channel(0) first = _open(url) st = _State(PoolStats(n, 0, 0, 0, 0, 0, 0, 0, 0, 0), time.mono_ms()) if first.live: st.stats.open = 1 st.stats.opened = 1 ok = first.live free.send(first) for i in 1..n: free.send(_no_conn()) key = "pg.tx." + crypto.hex(crypto.random_bytes(8)) shards: [shared[_Shard]] = [] for i in 0.._SHARDS: shards.push(task.shared(_Shard({}, 0, 0, 0, 0))) return Pool(url, n, free, task.shared(st), shards, key, ok, _ms(_param(url, "pool_timeout"), 30000), _ms(_param(url, "max_lifetime"), 3600000), _ms(_param(url, "max_idle"), 1800000)) fn connect(url: str) Pool: return pool(url, 1) fn ok(p: Pool) bool: return p.first_ok # the counters, and how many connections are idle and in use right now fn stats(p: Pool) PoolStats: idle = 0 for i in 0..len(p.free): c = _no_conn() if not p.free.try_recv(&c): break if c.live: idle += 1 p.free.send(c) s = p.state.get().stats for sh in p.shards: w = sh.get() s.waiting += w.waiting s.waits += w.waits s.wait_ms += w.wait_ms s.timeouts += w.timeouts s.idle = idle s.in_use = max(s.open - idle, 0) return s # why the last call on this task failed ("" after a success) fn error() str: return task.local_get("pg.error") fn _set_err(msg: str): if msg != "" or task.local_get("pg.error") != "": task.local_set("pg.error", msg) # The connection for a statement: the task's own (in a transaction), or one from the pool # that is still good (not broken, too old, idle too long, or closed by the server). Not # live: none (pg.error() says why). fn _take(p: Pool) Conn: if task.local_get(p.key) != "": # this task holds one, in a transaction tid = task.id() mine = _shard(p, tid).update(fn(s) _held(&s, tid)) if mine is Some(held): return held c = _no_conn() if not p.free.try_recv(&c): got = _wait(p) if got is Some(w): c = w else: return _no_conn() closed = 0 if c.live and (c.broken or _stale(p, c)): _close_conn(c) closed = 1 idled = p.idle_ms > 0 and time.mono_ms() - c.used_ms > p.idle_ms c = _no_conn() if idled: prune(p) # (the others have likely idled too) if c.live: return c c = _open(p.url) opened = 1 if c.live else 0 p.state.update(fn(s) _count(&s, opened - closed, opened, closed)) if not c.live: p.free.send(_no_conn()) return c fn _held(s: mut _Shard, tid: int) Option[Conn]: if not (tid in s.bound): return None c = s.bound[tid] s.bound.remove(tid) return Some(c) fn _count(s: mut _State, open: int, opened: int, closed: int) int: s.stats.open += open s.stats.opened += opened s.stats.closed += closed return 0 # every connection is in use: wait for one, up to pool_timeout fn _wait(p: Pool) Option[Conn]: sh = _shard(p, task.id()) sh.update(fn(s) _waiting(&s, 1, 0, false)) t0 = time.mono_ms() c = _no_conn() got = true if p.timeout_ms > 0: got = p.free.recv_timeout(&c, p.timeout_ms) else: c = p.free.recv() waited = time.mono_ms() - t0 sh.update(fn(s) _waiting(&s, -1, waited, not got)) if not got: busy = "the pool's one connection is" if p.size == 1 else f"all {p.size} connections are" _set_err(f"pg: no connection free after {p.timeout_ms} ms: {busy} in use (a bigger pool, shorter transactions, or a longer ?pool_timeout=)") return None return Some(c) fn _waiting(s: mut _Shard, now: int, waited: int, gave_up: bool) int: s.waiting += now if now < 0: s.waits += 1 s.wait_ms += waited if gave_up: s.timeouts += 1 return 0 fn _stale(p: Pool, c: Conn) bool: now = time.mono_ms() jitter = c.born_ms % 10 * (p.lifetime_ms / 100) # (up to 9% early: not all at once) if p.lifetime_ms > 0 and now - c.born_ms > p.lifetime_ms - jitter: return true idle = now - c.used_ms if p.idle_ms > 0 and idle > p.idle_ms: return true return idle >= 100 and c.sock.pending() # the server closed it (a restart, a kill) # after a statement: keep the connection for the task while a transaction is open fn _give_back(p: Pool, c: Conn): if not c.live: return if c.broken: _close_conn(c) p.state.update(fn(s) _count(&s, -1, 0, 1)) p.free.send(_no_conn()) _unmark(p) return if c.txs != 73: # in a transaction ('T', or 'E': failed) tid = task.id() _shard(p, tid).update(fn(s) _hold(&s, tid, c)) task.local_set(p.key, "1") return _unmark(p) back = c back.used_ms = time.mono_ms() back.uses += 1 p.free.send(back) if p.idle_ms > 0 and back.uses % 256 == 0: # now and then: close idle connections every = min(p.idle_ms / 2, 60000) now = back.used_ms if p.state.update(fn(s) _due(&s, now, every)): prune(p) fn _unmark(p: Pool): if task.local_get(p.key) != "": task.local_set(p.key, "") fn _hold(s: mut _Shard, tid: int, c: Conn) int: s.bound[tid] = c return 0 fn _due(s: mut _State, now: int, every: int) bool: if now - s.pruned < every: return false s.pruned = now return true # Closes the connections idle longer than max_idle, so a pool shrinks after a busy spell. # The pool does this by itself now and then, as connections come back. fn prune(p: Pool): if p.idle_ms <= 0: return now = time.mono_ms() closed = 0 for i in 0..len(p.free): c = _no_conn() if not p.free.try_recv(&c): break if c.live and now - c.used_ms > p.idle_ms: _close_conn(c) closed += 1 c = _no_conn() p.free.send(c) if closed > 0: p.state.update(fn(s) _count(&s, -closed, 0, closed)) # closes the idle connections (the pool opens new ones when it needs them again) fn close(p: Pool): closed = 0 for i in 0..len(p.free): c = _no_conn() if not p.free.try_recv(&c): break if c.live: _close_conn(c) closed += 1 p.free.send(_no_conn()) if closed > 0: p.state.update(fn(s) _count(&s, -closed, 0, closed)) # a URL's ?name=value ("" if it has none) fn _param(url: str, name: str) str: q = url.find("?") if q < 0: return "" for kv in url[q + 1..].split("&"): if kv.starts_with(name + "="): return kv[len(name) + 1..] return "" # "5s", "500ms", "30m", "1h", "90" (seconds), "0" (off) -> milliseconds; dflt if empty or not one fn _ms(text: str, dflt: int) int: units = [("ms", 1), ("s", 1000), ("m", 60000), ("h", 3600000)] for u in units: if text.ends_with(u.0) and len(text) > len(u.0): n = text[..len(text) - len(u.0)].to_float(-1.0) if n >= 0.0: return int(n * float(u.1)) n = text.to_float(-1.0) if n >= 0.0 and text != "": return int(n * 1000.0) return dflt # ---------------------------------------------------------------- statements fn exec(p: Pool, sql: str, params: [str]) int: c = _take(p) if not c.live: return -1 r = _run(&c, sql, params, 0) _give_back(p, c) return r.affected if r.ok else -1 fn _rows(p: Pool, sql: str, params: [str], mode: int) str: c = _take(p) if not c.live: return "null" if mode == 3 else ("" if mode == 4 else "[]") r = _run(&c, sql, params, mode) _give_back(p, c) if not r.ok: return "null" if mode == 3 else ("" if mode == 4 else "[]") return r.out fn query_json(p: Pool, sql: str, params: [str]) str: return _rows(p, sql, params, 1) fn query_row_json(p: Pool, sql: str, params: [str]) str: return _rows(p, sql, params, 3) fn query(p: Pool, sql: str, params: [str]) [{str: str}]: rows: [{str: str}] = [] json.decode(_rows(p, sql, params, 2), &rows) return rows fn begin(p: Pool) bool: return exec(p, "BEGIN", []) >= 0 fn commit(p: Pool) bool: return exec(p, "COMMIT", []) >= 0 fn rollback(p: Pool) bool: return exec(p, "ROLLBACK", []) >= 0 # ---------------------------------------------------------------- with Result # The same queries, returning Result: rows decoded straight into your struct type, and the # server's message as the error ("pg: query failed: ERROR 42P01: relation \"x\" does not exist"). # users: [User] = db.fetch("SELECT id, name FROM users WHERE team = $1", [team])? # u = db.fetch_one[User]("SELECT id, name FROM users WHERE id = $1", [id])? # Option[User] # n = db.run("DELETE FROM sessions WHERE expires < now()", [])? # rows affected fn _fail(what: str) str: return "pg: " + what + ": " + error() fn fetch[T](p: Pool, sql: str, params: [str]) Result[[T], str]: raw = _rows(p, sql, params, 4) # the rows as the server sent them if error() != "": return Err(_fail("query failed")) rows: [T] = [] if sql.decode_rows(raw, &rows) < 0: # decoded by code made for T why = sql.decode_error() # (which value didn't fit which field) if why != "": return Err("pg: query failed: " + why) return Err("pg: query failed: the rows don't fit the type (check the column names and types)") return Ok(rows) fn fetch_one[T](p: Pool, sql: str, params: [str]) Result[Option[T], str]: rows: [T] = fetch(p, sql, params)? none: Option[T] = None if len(rows) == 0: return Ok(none) return Ok(Some(rows[0])) fn run(p: Pool, sql: str, params: [str]) Result[int, str]: n = exec(p, sql, params) if n < 0: return Err(_fail("statement failed")) return Ok(n) # several statements in one string (a schema file, a migration), run with the simple query # protocol: in one implicit transaction unless the script has its own BEGIN / COMMIT, and # with no $1 parameters. The rows affected, added up; the first error stops the script. fn run_script(p: Pool, sql: str) Result[int, str]: c = _take(p) if not c.live: return Err(_fail("script failed")) r = _script(&c, sql) _give_back(p, c) if not r.ok: return Err(_fail("script failed")) return Ok(r.affected) # ---------------------------------------------------------------- describing a statement # What the server knows about a statement, without running it: the types of its $1, $2, ... # parameters and of the columns it returns (the base for generating typed query code). # s = db.describe("SELECT id, email FROM users WHERE id = $1")? # s.params ["int8"] # s.columns[1] Column{name: "email", type_name: "text", table: "users", not_null: false, ...} struct Column: name: str type_name: str # pg_type.typname: "int8", "text", "timestamptz", "_int4" (int4[]), ... type_oid: int kind: str # pg_type.typtype: "b" base, "e" enum, "d" domain, "c" composite, ... base_type: str # a domain's underlying type ("" otherwise) table: str # the table (or view) the column comes from; "" for an expression not_null: bool # a table column declared NOT NULL (an outer join can still make it null) struct Statement: params: [str] # the type names of $1, $2, ... param_oids: [int] param_kinds: [str] # ... and their pg_type.typtype ("e": an enum) columns: [Column] # [] for a statement that returns no rows struct _Type: oid: int name: str kind: str base: str struct _Attr: rel: int num: int notnull: bool relname: str fn describe(p: Pool, sql: str) Result[Statement, str]: c = _take(p) if not c.live: return Err(_fail("describe failed")) got = _describe_raw(&c, sql) _give_back(p, c) raw = got.map_err(fn(e) "pg: describe failed: " + e.replace("pg: ", ""))? oids: [int] = [] rels: [int] = [] for o in raw.params: oids.push(o) for c in raw.columns: oids.push(c.oid) if c.table != 0: rels.push(c.table) types: {int: _Type} = {} ts = fetch[_Type](p, "SELECT t.oid::int8 AS oid, t.typname::text AS name, t.typtype::text AS kind, coalesce(b.typname::text, '') AS base FROM pg_type t LEFT JOIN pg_type b ON b.oid = t.typbasetype AND t.typtype = 'd' WHERE t.oid = ANY($1::oid[])", [_oid_array(oids)])? for t in ts: types[t.oid] = t attrs: {str: _Attr} = {} if len(rels) > 0: as_ = fetch[_Attr](p, "SELECT a.attrelid::int8 AS rel, a.attnum::int8 AS num, a.attnotnull AS notnull, c.relname::text AS relname FROM pg_attribute a JOIN pg_class c ON c.oid = a.attrelid WHERE a.attrelid = ANY($1::oid[]) AND a.attnum > 0", [_oid_array(rels)])? for a in as_: attrs[f"{a.rel}.{a.num}"] = a out = Statement([], [], [], []) for o in raw.params: out.params.push(_type_of(types, o).name) out.param_kinds.push(_type_of(types, o).kind) out.param_oids.push(o) for c in raw.columns: t = _type_of(types, c.oid) col = Column(c.name, t.name, c.oid, t.kind, t.base, "", false) key = f"{c.table}.{c.column}" if c.table != 0 and key in attrs: col.table = attrs[key].relname col.not_null = attrs[key].notnull out.columns.push(col) return Ok(out) fn _type_of(types: {int: _Type}, oid: int) _Type: if oid in types: return types[oid] return _Type(oid, "unknown", "", "") fn _oid_array(xs: [int]) str: return "{" + ",".join(xs.map(fn(x) str(x))) + "}" # A parameter that is SQL NULL: pass pg.NULL in params (["x", pg.NULL]). It is the one-byte # string "\x00", which PostgreSQL text can never contain. const NULL = "\x00" # Arrays as parameters, in PostgreSQL's text form: `WHERE id = ANY($1)` with [pg.array_int(ids)]. # (Rows read arrays back into [int], [float], [bool] and [str] fields by themselves.) fn array_int(xs: [int]) str: return "{" + ",".join(xs.map(fn(x) str(x))) + "}" fn array_float(xs: [float]) str: return "{" + ",".join(xs.map(fn(x) str(x))) + "}" fn array_bool(xs: [bool]) str: return "{" + ",".join(xs.map(fn(x) "t" if x else "f")) + "}" fn array_str(xs: [str]) str: return "{" + ",".join(xs.map(fn(x) "\"" + x.replace("\\", "\\\\").replace("\"", "\\\"") + "\"")) + "}" ``` ## deps/pg/wire.tn ```tern # pg: the PostgreSQL wire protocol (version 3), in Tern (pg.tn builds the API on it): # connecting (TLS through SSLRequest; SCRAM-SHA-256 or password logins), statements through # the extended protocol with each connection's prepared statements kept by their SQL, scripts # through the simple protocol, and replies read into what the API returns. # a statement prepared on a connection struct Stmt: name: str # "t3" ("": unnamed, prepared again each time) cols: [str] # the columns it returns keys: [str] # ... as JSON keys: "\"id\":" types: [int] # ... and their type OIDs head: str # Bind's constant start: portal, name, parameter formats ("": not prepared) fmts: str # ... and its result formats for pg.fetch (numbers in binary; "": all text) struct Conn: sock: net.Conn live: bool # false: no connection (an empty slot in a pool) buf: str # bytes read and not used yet, from `at` on at: int ms: int # the message just read: where its body starts in buf, ml: int # ... its length, mt: int # ... and its type ('D', 'Z', ...) stmts: {str: Stmt} # the statements prepared here, by their SQL txs: int # from ReadyForQuery: 'I' idle, 'T' in a transaction, 'E' a failed one broken: bool no_cache: bool # ?statement_cache=off: unnamed statements only (PgBouncer) born_ms: int # opened, and last used (time.mono_ms): for the pool used_ms: int uses: int # statements run (the pool looks at idle connections every so often) # what a statement's reply comes to struct Reply: ok: bool affected: int # rows changed (or returned) out: str # the rows: JSON (modes 1-3) or raw (mode 4) struct Url: user: str password: str host: str port: int db: str ssl: str app: str cache: str fn _no_conn() Conn: return Conn(net.Conn(-1), false, "", 0, 0, 0, 0, {}, 73, true, false, 0, 0, 0) fn _fail_reply(msg: str) Reply: _set_err(msg) return Reply(false, -1, "") # ---------------------------------------------------------------- bytes fn _u32(n: int) str: return chr((n >> 24) & 255) + chr((n >> 16) & 255) + chr((n >> 8) & 255) + chr(n & 255) fn _u16(n: int) str: return chr((n >> 8) & 255) + chr(n & 255) fn _le32(n: int) str: # (the raw rows' header is read with memcpy: native order) return chr(n & 255) + chr((n >> 8) & 255) + chr((n >> 16) & 255) + chr((n >> 24) & 255) fn _msg(t: str, body: str) str: return t + _u32(len(body) + 4) + body fn _be32(s: str, i: int) int: return (s.byte(i) << 24) | (s.byte(i + 1) << 16) | (s.byte(i + 2) << 8) | s.byte(i + 3) fn _be16(s: str, i: int) int: return (s.byte(i) << 8) | s.byte(i + 1) fn _i32(s: str, i: int) int: v = _be32(s, i) return v - 4294967296 if v >= 2147483648 else v # the NUL-terminated string at s[i..] (before e) fn _cstr(s: str, i: int, e: int) str: j = i while j < e and s.byte(j) != 0: j += 1 return s[i..j] # ---------------------------------------------------------------- reading and writing # at least n bytes from c.at in c.buf fn _read(c: mut Conn, n: int) bool: while len(c.buf) - c.at < n: got = c.sock.refill(&c.buf, c.at, 65536) # (what was at c.at is now at 0) c.ms -= c.at c.at = 0 if got == 0: c.broken = true why = net.error() _set_err("pg: connection lost" + (" (" + why + ")" if why != "" else "")) return false return true # the next message (notices, parameter changes and notifications skipped): c.mt, c.ms, c.ml fn _next(c: mut Conn) bool: while true: c.at = c.ms + c.ml if len(c.buf) - c.at < 5 and not _read(&c, 5): return false n = _be32(c.buf, c.at + 1) if n < 4 or n > 1073741824: c.broken = true _set_err("pg: protocol error (bad message length)") return false c.ms = c.at + 5 c.ml = 0 if len(c.buf) - c.at < n + 1 and not _read(&c, n + 1): return false c.mt = c.buf.byte(c.at) c.ms = c.at + 5 c.ml = n - 4 if not (c.mt == 78 or c.mt == 83 or c.mt == 65): return true return false fn _send(c: mut Conn, out: str) bool: if c.sock.write(out): return true c.broken = true why = net.error() _set_err("pg: connection lost" + (" (" + why + ")" if why != "" else "")) return false # an ErrorResponse: "ERROR 42P01: relation \"x\" does not exist (DETAIL) hint: HINT" fn _server_error(c: Conn) str: sev = "ERROR" code = "" msg = "" detail = "" hint = "" i = c.ms e = c.ms + c.ml while i < e and c.buf.byte(i) != 0: f = c.buf.byte(i) v = _cstr(c.buf, i + 1, e) i += len(v) + 2 if f == 83: sev = v elif f == 67: code = v elif f == 77: msg = v elif f == 68: detail = v elif f == 72: hint = v out = f"{sev} {code}: {msg}" if detail != "": out += f" ({detail})" if hint != "": out += " hint: " + hint return out # ---------------------------------------------------------------- connecting # postgres://user:password@host:port/dbname?sslmode=require&application_name=x&statement_cache=off fn _parse_url(url: str) Result[Url, str]: rest = "" if url.starts_with("postgresql://"): rest = url[13..] elif url.starts_with("postgres://"): rest = url[11..] else: return Err("pg: not a postgres:// url") u = Url("postgres", "", "localhost", 5432, "", "prefer", "tern", "") query = "" q = rest.find("?") if q >= 0: query = rest[q + 1..] rest = rest[..q] hostpart = rest slash = rest.find("/") if slash >= 0: u.db = _unescape(rest[slash + 1..]) hostpart = rest[..slash] at = _last(hostpart, "@") if at >= 0: cred = hostpart[..at] hostpart = hostpart[at + 1..] colon = cred.find(":") if colon >= 0: u.user = _unescape(cred[..colon]) u.password = _unescape(cred[colon + 1..]) else: u.user = _unescape(cred) if hostpart.starts_with("["): rb = hostpart.find("]") if rb < 0: return Err("pg: not a postgres:// url") u.host = hostpart[1..rb] if hostpart[rb + 1..].starts_with(":"): u.port = hostpart[rb + 2..].to_int(5432) else: colon = _last(hostpart, ":") if colon >= 0: u.port = hostpart[colon + 1..].to_int(5432) hostpart = hostpart[..colon] if hostpart != "": u.host = _unescape(hostpart) if u.db == "": u.db = u.user for kv in query.split("&"): eq = kv.find("=") if eq < 0: continue v = _unescape(kv[eq + 1..]) k = kv[..eq] if k == "sslmode": u.ssl = v elif k == "application_name": u.app = v elif k == "statement_cache": u.cache = v return Ok(u) fn _last(s: str, sub: str) int: # (sub: one byte) b = sub.byte(0) i = len(s) - 1 while i >= 0 and s.byte(i) != b: i -= 1 return i fn _unescape(s: str) str: # %40 -> @ if not s.contains("%"): return s out = "" i = 0 while i < len(s): if s[i] == "%" and i + 2 < len(s) + 0 and _hex(s.byte(i + 1)) >= 0 and _hex(s.byte(i + 2)) >= 0: out += chr(_hex(s.byte(i + 1)) * 16 + _hex(s.byte(i + 2))) i += 3 else: out += s[i] i += 1 return out fn _hex(c: int) int: if c >= 48 and c <= 57: return c - 48 if c >= 97 and c <= 102: return c - 87 if c >= 65 and c <= 70: return c - 55 return -1 # a new connection, logged in and ready; or not live, and pg.error() says why fn _open(url: str) Conn: r = _open_r(url) if r is Err(e): _set_err(e) return _no_conn() _set_err("") return r.unwrap() fn _open_r(url: str) Result[Conn, str]: u = _parse_url(url)? sock = net.connect(u.host, u.port) if not sock.ok(): return Err("pg: " + net.error()) now = time.mono_ms() c = Conn(sock, true, "", 0, 0, 0, 0, {}, 73, false, u.cache == "off", now, now, 0) r = _login(&c, u) if r is Err(e): sock.close() return Err(e) return Ok(c) fn _login(c: mut Conn, u: Url) Result[int, str]: if not (u.ssl in ["disable", "prefer", "allow"]): if not _send(&c, _u32(8) + _u32(80877103)): # SSLRequest return Err(error()) answer = c.sock.read_exact(1) if answer != "S": return Err("pg: the server doesn't accept TLS connections") if not c.sock.starttls(u.host, u.ssl == "verify-full"): return Err("pg: " + net.error()) body = _u32(196608) + "user\x00" + u.user + "\x00database\x00" + u.db + "\x00application_name\x00" + u.app + "\x00client_encoding\x00UTF8\x00\x00" if not _send(&c, _u32(len(body) + 4) + body): return Err(error()) while true: if not _next(&c): return Err(error()) if c.mt == 69: return Err(_server_error(c)) if c.mt == 90: c.txs = c.buf.byte(c.ms) return Ok(0) if c.mt != 82 or c.ml < 4: continue # BackendKeyData, ... kind = _be32(c.buf, c.ms) if kind == 0: continue # AuthenticationOk if kind == 3: # a password in clear (over TLS, one hopes) if not _send(&c, _msg("p", u.password + "\x00")): return Err(error()) elif kind == 10: mechs = c.buf[c.ms + 4..c.ms + c.ml] if not mechs.contains("SCRAM-SHA-256\x00"): return Err("pg: the server wants a SASL mechanism other than SCRAM-SHA-256") _scram(&c, u.password)? else: return Err(f"pg: unsupported authentication method {kind} (use scram-sha-256 or password)") return Err("pg: login failed") # SCRAM-SHA-256 (RFC 5802 / 7677), what PostgreSQL uses by default fn _scram(c: mut Conn, password: str) Result[int, str]: nonce = crypto.base64(crypto.random_bytes(18)) first_bare = "n=,r=" + nonce first = "n,," + first_bare if not _send(&c, _msg("p", "SCRAM-SHA-256\x00" + _u32(len(first)) + first)): return Err(error()) server_first = _sasl_reply(&c, 11)? rnonce = "" salt64 = "" iters = 0 for part in server_first.split(","): if part.starts_with("r="): rnonce = part[2..] elif part.starts_with("s="): salt64 = part[2..] elif part.starts_with("i="): iters = part[2..].to_int(0) if not rnonce.starts_with(nonce): return Err("pg: SCRAM: bad server message") salt = crypto.base64_decode(salt64) if salt == "" or iters < 1: return Err("pg: SCRAM: bad salt") salted = crypto.pbkdf2_sha256(password, salt, iters, 32) client_key = crypto.hmac_sha256(salted, "Client Key") final_bare = "c=biws,r=" + rnonce auth = first_bare + "," + server_first + "," + final_bare sig = crypto.hmac_sha256(crypto.sha256(client_key), auth) proof = "" for i in 0..32: proof += chr(client_key.byte(i) ^ sig.byte(i)) if not _send(&c, _msg("p", final_bare + ",p=" + crypto.base64(proof))): return Err(error()) server_final = _sasl_reply(&c, 12)? want = "v=" + crypto.base64(crypto.hmac_sha256(crypto.hmac_sha256(salted, "Server Key"), auth)) if not server_final.starts_with(want): return Err("pg: SCRAM: the server's signature is wrong (not the real server?)") return Ok(0) fn _sasl_reply(c: mut Conn, want: int) Result[str, str]: if not _next(&c): return Err(error()) if c.mt == 69: return Err(_server_error(c)) if c.mt != 82 or c.ml < 4 or _be32(c.buf, c.ms) != want: return Err("pg: SCRAM: unexpected reply") return Ok(c.buf[c.ms + 4..c.ms + c.ml]) fn _close_conn(c: Conn): if c.live and not c.broken: c.sock.write("X\x00\x00\x00\x04") # Terminate if c.live: c.sock.close() # ---------------------------------------------------------------- statements # Types a prepared statement's rows bring back in binary, for pg.fetch (mode 4): numbers # and booleans, which the server then needn't format nor the decoder parse. fn _binary(oid: int) bool: return oid == 16 or oid == 20 or oid == 21 or oid == 23 or oid == 26 or oid == 700 or oid == 701 # Runs sql with params. mode 0: the rows affected; 1: the rows as JSON (typed); 2: as JSON, # every value a string (NULL: ""); 3: like 1, only the first row, an object ("null": none); # 4: raw, for the decoders the compiler writes (sql.decode_rows). fn _run(c: mut Conn, sql: str, params: [str], mode: int) Reply: if c.broken: return _fail_reply("pg: the connection is broken") st = c.stmts.get(sql, Stmt("", [], [], [], "", "")) cached = st.head != "" pre = "" if not cached: name = "t" + str(len(c.stmts)) if not c.no_cache and len(c.stmts) < 256 else "" st = Stmt(name, [], [], [], "\x00" + name + "\x00\x00\x00", "") pre = _msg("P", name + "\x00" + sql + "\x00\x00\x00") + _msg("D", "S" + name + "\x00") binary = mode == 4 and cached and st.fmts != "" fm = st.fmts if binary else "\x00\x00" vals: [str] = [] # the parameters, as Bind has them size = 0 for v in params: if v == "\x00": vals.push("\xff\xff\xff\xff") # pg.NULL size += 4 else: vals.push(_u32(len(v))) vals.push(v) size += 4 + len(v) out = pre + "B" + _u32(6 + len(st.head) + size + len(fm)) + st.head + _u16(len(params)) + "".join(vals) + fm + "E\x00\x00\x00\x09\x00\x00\x00\x00\x00S\x00\x00\x00\x04" if not _send(&c, out): return Reply(false, -1, "") res = "[" if mode == 1 or mode == 2 else "" rows = 0 affected = 0 err = "" parsed = cached described = cached while _next(&c): t = c.mt if t == 90: # ReadyForQuery c.txs = c.buf.byte(c.ms) if not cached and parsed and st.name != "" and described: c.stmts[sql] = st # kept, even if running it failed if err != "": return _fail_reply(err) if mode == 1 or mode == 2: res += "]" elif mode == 3 and rows == 0: res = "null" elif mode == 4 and rows == 0: res = _raw_header(st, binary) _set_err("") return Reply(true, affected, res) if t == 69: if err == "": err = _server_error(c) elif err != "": pass elif t == 49: # ParseComplete parsed = true elif t == 84 and not cached: # RowDescription st = _row_description(c, st.name, st.head) described = true elif t == 110 and not cached: # NoData described = true elif t == 68: # DataRow if mode == 4: if rows == 0: res = _raw_header(st, binary) # this row and the ones right after it in the buffer, copied as one block # (whole messages: the decoder steps over each one's type and length) end = c.ms + c.ml k = 1 while end + 5 <= len(c.buf) and c.buf.byte(end) == 68: m = _be32(c.buf, end + 1) if end + 1 + m > len(c.buf): break end += 1 + m k += 1 res += c.buf[c.ms - 5..end] rows += k c.ml = end - c.ms # (the next message is after the block) continue rows += 1 if mode != 0 and not (mode == 3 and rows > 1): if rows > 1: res += "," res += _json_row(c, st, mode) elif t == 67: # CommandComplete: "UPDATE 2" affected = _tag_count(_cstr(c.buf, c.ms, c.ms + c.ml), rows) return Reply(false, -1, "") fn _row_description(c: Conn, name: str, head: str) Stmt: st = Stmt(name, [], [], [], head, "") n = _be16(c.buf, c.ms) i = c.ms + 2 e = c.ms + c.ml for k in 0..n: col = _cstr(c.buf, i, e) i += len(col) + 1 st.cols.push(copy(col)) st.keys.push(json_quote(col) + ":") st.types.push(_be32(c.buf, i + 6)) i += 18 fm = _u16(n) any = false for t in st.types: fm += "\x00\x01" if _binary(t) else "\x00\x00" any = any or _binary(t) if any: st.fmts = fm return st fn _tag_count(tag: str, rows: int) int: sp = _last(tag, " ") return rows if sp < 0 else tag[sp + 1..].to_int(rows) # a DataRow as a JSON object fn _json_row(c: Conn, st: Stmt, mode: int) str: out = "{" n = min(_be16(c.buf, c.ms), len(st.keys)) i = c.ms + 2 for k in 0..n: vlen = _i32(c.buf, i) i += 4 if k > 0: out += "," out += st.keys[k] if vlen < 0: out += "\"\"" if mode == 2 else "null" continue v = c.buf[i..i + vlen] i += vlen out += _json_value(v, st.types[k], mode) return out + "}" fn _json_value(v: str, oid: int, mode: int) str: if mode != 2: if oid == 16: return "true" if v == "t" else "false" if oid == 20 or oid == 21 or oid == 23 or oid == 26: return v if (oid == 700 or oid == 701 or oid == 1700) and len(v) > 0 and not v.ends_with("y") and (v[0] == "-" or (v.byte(0) >= 48 and v.byte(0) <= 57)): return v return json_quote(v) # the raw rows' header (library/std/sql.tn): the columns and how their values are written; # after it come the DataRow messages, whole (they're the rows sql.decode_rows reads) fn _raw_header(st: Stmt, binary: bool) str: out = _le32(len(st.cols)) for k, col in st.cols: out += _le32(_format(st.types[k]) if binary else sql.TEXT) + _le32(len(col)) + col return out # how a binary field of PostgreSQL type oid is written, as std sql names it fn _format(oid: int) int: if oid == 16: return sql.BOOL if oid == 20 or oid == 21 or oid == 23: return sql.INT if oid == 26: return sql.UINT if oid == 700 or oid == 701: return sql.FLOAT return sql.TEXT # Several statements in one string, through the simple protocol: in one implicit transaction # unless the script has its own BEGIN / COMMIT; the first error stops the rest. The rows # affected, added up. fn _script(c: mut Conn, sql: str) Reply: if c.broken: return _fail_reply("pg: the connection is broken") if not _send(&c, _msg("Q", sql + "\x00")): return Reply(false, -1, "") total = 0 err = "" while _next(&c): t = c.mt if t == 90: c.txs = c.buf.byte(c.ms) if err != "": return _fail_reply(err) _set_err("") return Reply(true, total, "") if t == 69 and err == "": err = _server_error(c) elif t == 67: tag = _cstr(c.buf, c.ms, c.ms + c.ml) sp = _last(tag, " ") if sp >= 0: total += tag[sp + 1..].to_int(0) elif t == 71: # CopyInResponse: a script can't send data if not _send(&c, _msg("f", "COPY FROM STDIN is not supported in a script\x00")): return Reply(false, -1, "") return Reply(false, -1, "") struct _RawCol: name: str oid: int table: int column: int struct _Raw: params: [int] columns: [_RawCol] # what the server knows about a statement, without running it (Parse and Describe, unnamed) fn _describe_raw(c: mut Conn, sql: str) Result[_Raw, str]: if c.broken: return Err("pg: the connection is broken") out = _msg("P", "\x00" + sql + "\x00\x00\x00") + _msg("D", "S\x00") + "S\x00\x00\x00\x04" if not _send(&c, out): return Err(error()) raw = _Raw([], []) err = "" while _next(&c): t = c.mt if t == 90: c.txs = c.buf.byte(c.ms) if err != "": return Err(err) return Ok(raw) if t == 69 and err == "": err = _server_error(c) elif t == 116 and err == "": # ParameterDescription for k in 0.._be16(c.buf, c.ms): raw.params.push(_be32(c.buf, c.ms + 2 + 4 * k)) elif t == 84 and err == "": i = c.ms + 2 e = c.ms + c.ml for k in 0.._be16(c.buf, c.ms): col = _cstr(c.buf, i, e) i += len(col) + 1 raw.columns.push(_RawCol(copy(col), _be32(c.buf, i + 6), _be32(c.buf, i), _be16(c.buf, i + 4))) i += 18 return Err(error()) ``` ## deps/queue/queue.tn ```tern # queue: a client for ternq, the message queue broker (examples/queue/). # # import queue # c = queue.connect("127.0.0.1", 7070) # check c.ok # c.publish("jobs", "hello") # true once the broker has it on disk # s = c.subscribe("jobs", "workers", 64) # group "workers", up to 64 unacked at once # m = queue.empty_msg() # while s.next(&m): # print(m.body) # c.ack(m) # or c.nack(m) to have it redelivered # # queue.connect(host: str, port: int) Client one connection, shared by any number of tasks # queue.connect_many(addrs: [str]) Client several brokers ("host:port"); each queue lives # on one, chosen by a hash of its name (sharding). # An entry can list a primary and its followers, # "h1:7070,h2:7070": the client moves to whichever # of them is the primary (failover) # queue.connect_auth(addrs: [str], token: str) Client the same, for brokers started with --token # c.shard_of(q: str) int which of them holds q # queue.empty_msg() Msg a Msg to receive into # c.ok bool the first connect worked (it reconnects by itself later) # c.publish(q: str, body: str) bool waits for the broker's ack (false: client closed) # c.publish_delayed(q: str, body: str, delay_ms: int) bool deliver no sooner than delay_ms from now # c.publish_batch(q: str, bodies: [str]) bool one frame, one ack for all of them # c.publish_async(q: str, body: str) chan[bool] don't wait: recv() the result later (pipelining) # c.subscribe(q: str, group: str, prefetch: int) Sub each group gets every message; the # consumers in a group share them. prefetch 0: pull only # s.next(m: mut Msg) bool waits for the next message (false: client closed) # s.next_timeout(m: mut Msg, ms: int) bool false if none came within ms # s.msgs chan[Msg] the subscription's messages, as a channel # c.fetch(s: Sub, max: int) [Msg] pull up to max messages now (any subscription) # c.fetch_wait(s: Sub, max: int, ms: int) [Msg] the same, but with none there, wait up to ms for some # c.ack(m: Msg) c.nack(m: Msg) acks are batched into one frame per burst # c.unsubscribe(s: Sub) its unacked messages go back to the group # c.stats(q: str) str JSON: depth, in-flight, rates, per group # c.queues() str JSON array of queue names # c.shutdown_broker() ask the broker to stop gracefully (admin) # c.close() # # Delivery is at least once: a message is redelivered if it isn't acked within the # broker's visibility timeout, if it's nacked, or if its consumer disconnects. After a # lost connection the client reconnects (with backoff), resends every publish that # wasn't acked yet, and subscribes again, so a publish may arrive twice. With a failover # list it tries the next address after a lost connection, and it tells every broker the # highest epoch it has seen, so a replaced primary that is still running stops rather # than taking writes (see examples/queue/README.md). # # Inside: one task owns the connection. Callers send it requests on a channel; it # writes whatever has piled up with one write, and a reader task parses what the # broker sends back (messages go straight to their subscription's channel). import net import task import time struct Msg: sid: int seq: int attempt: int body: str shard: int # which broker (connect_many) struct Client: ok: bool shards: [chan[Req]] # one connection task per broker struct Sub: sid: int shard: int msgs: chan[Msg] enum Req: Publish(queue: str, body: str, delay: int, reply: chan[bool]) PublishBatch(queue: str, bodies: [str], reply: chan[bool]) Subscribe(queue: str, group: str, prefetch: int, msgs: chan[Msg], reply: chan[int]) Unsubscribe(sid: int) Ack(sid: int, seq: int, ok: bool) Fetch(sid: int, max: int, wait: int, reply: chan[int]) Query(line: str, reply: chan[str]) Raw(line: str) Close Reply(gen: int, id: int, ok: bool) # from the reader Json(gen: int, id: int, text: str) Fetched(gen: int, sid: int, n: int) Lost(gen: int) Deliver(gen: int, m: Msg) Epoch(gen: int, epoch: int) struct Pending: # a request waiting for its reply frame: str ok: chan[bool] txt: chan[str] kind: int # 0 publish, 1 query struct SubInfo: queue: str group: str prefetch: int msgs: chan[Msg] struct Addr: host: str port: int struct State: addrs: [Addr] # a primary and its followers cur: int # the one we're connected to token: str epoch: int # the highest the brokers told us shard: int conn: net.Conn gen: int next_id: int pend: {int: Pending} subs: {int: SubInfo} fetches: {int: [chan[int]]} out: str acks: {int: str} closed: bool up: bool fn empty_msg() Msg: return Msg(0, 0, 0, "", 0) fn connect(host: str, port: int) Client: return connect_many([f"{host}:{port}"]) fn connect_many(addrs: [str]) Client: return connect_auth(addrs, "") # several brokers: each queue lives on one of them, picked by hashing its name # (every client given the same list agrees on where each queue is) fn connect_auth(addrs: [str], token: str) Client: shards: [chan[Req]] = [] ok = len(addrs) > 0 for i, entry in addrs: list = entry.split(",").map(fn(a) _addr(a.trim())) reqs: chan[Req] = task.channel(0) st = State(list, 0, token, 0, i, net.Conn(-1), 1, 1, {}, {}, {}, "", {}, false, true) if _dial(&st): task.spawn(fn() run(st, reqs)) else: ok = false shards.push(reqs) return Client(ok, shards) fn _addr(a: str) Addr: k = a.find(":") j = k while j >= 0: # the last ":" ("[::1]:7070" works too) k = j rest = a[k + 1..] j = rest.find(":") if j >= 0: j += k + 1 return Addr(a[..k].replace("[", "").replace("]", ""), a[k + 1..].to_int(7070)) # connect to the current address, or the next ones in turn: false if none answered fn _dial(s: mut State) bool: for i in 0..len(s.addrs): a = s.addrs[s.cur] c = net.connect(a.host, a.port) if c.ok(): s.conn = c if s.token != "" or len(s.addrs) > 1: c.write(f"I {s.epoch} {s.token}\n") return true s.cur = (s.cur + 1) % len(s.addrs) return false # which broker holds queue q: the one scoring highest for it (rendezvous hashing, so # adding a broker to the list moves only the queues that now score highest on it) fn shard_of(c: Client, q: str) int: best = 0 top = -1 for i in 0..len(c.shards): h = i + 1 for k in 0..len(q): h = (h * 31 + q.byte(k)) % 1000000007 h = (h * 2654435761 + i * 40503) % 1000000007 if h > top: top = h best = i return best fn _to(c: Client, q: str) chan[Req]: return c.shards[shard_of(c, q)] fn publish(c: Client, q: str, body: str) bool: return publish_async(c, q, body).recv() fn publish_delayed(c: Client, q: str, body: str, delay_ms: int) bool: r: chan[bool] = task.channel(1) _to(c, q).send(Req.Publish(q, body, delay_ms, r)) return r.recv() fn publish_async(c: Client, q: str, body: str) chan[bool]: r: chan[bool] = task.channel(1) _to(c, q).send(Req.Publish(q, body, 0, r)) return r fn publish_batch(c: Client, q: str, bodies: [str]) bool: r: chan[bool] = task.channel(1) _to(c, q).send(Req.PublishBatch(q, bodies, r)) return r.recv() fn subscribe(c: Client, q: str, group: str, prefetch: int) Sub: msgs: chan[Msg] = task.channel(0) r: chan[int] = task.channel(1) _to(c, q).send(Req.Subscribe(q, group, prefetch, msgs, r)) return Sub(r.recv(), shard_of(c, q), msgs) fn next(s: Sub, m: mut Msg) bool: return task.next(s.msgs, &m) fn next_timeout(s: Sub, m: mut Msg, ms: int) bool: return s.msgs.recv_timeout(&m, ms) fn fetch(c: Client, s: Sub, max: int) [Msg]: return fetch_wait(c, s, max, 0) fn fetch_wait(c: Client, s: Sub, max: int, ms: int) [Msg]: r: chan[int] = task.channel(1) c.shards[s.shard].send(Req.Fetch(s.sid, max, ms, r)) n = r.recv() out: [Msg] = [] m = empty_msg() for i in 0..n: if not task.next(s.msgs, &m): break out.push(copy(m)) return out fn ack(c: Client, m: Msg): c.shards[m.shard].send(Req.Ack(m.sid, m.seq, true)) fn nack(c: Client, m: Msg): c.shards[m.shard].send(Req.Ack(m.sid, m.seq, false)) fn unsubscribe(c: Client, s: Sub): c.shards[s.shard].send(Req.Unsubscribe(s.sid)) fn stats(c: Client, q: str) str: r: chan[str] = task.channel(1) _to(c, q).send(Req.Query("T {id} " + q, r)) return r.recv() fn queues(c: Client) str: out = "" for sh in c.shards: r: chan[str] = task.channel(1) sh.send(Req.Query("L {id}", r)) txt = r.recv() if len(txt) > 2: if out != "": out += "," out += txt[1..len(txt) - 1] return "[" + out + "]" fn shutdown_broker(c: Client): for sh in c.shards: sh.send(Req.Raw("X\n")) fn close(c: Client): for sh in c.shards: sh.send(Req.Close) # ---------------------------------------------------------------- the connection's task fn run(st: State, reqs: chan[Req]): s = st g = s.gen cn = s.conn task.spawn(fn() reader(cn, g, reqs)) r = Req.Close while not s.closed and task.next(reqs, &r): handle(&s, r, reqs) n = 0 while n < 1024 and reqs.try_recv(&r): handle(&s, r, reqs) n += 1 if not s.up and not s.closed: reconnect(&s, reqs) if s.closed: if s.up: s.conn.write(s.out + "Q\n") # the broker closes the connection; the reader ends break send_out(&s) for id, p in s.pend: if p.kind == 0: p.ok.send(false) else: p.txt.send("") for sid, si in s.subs: task.close(si.msgs) for sid, fs in s.fetches: for f in fs: f.send(0) fn send_out(s: mut State): out = s.out for sid, seqs in s.acks: out += seqs + "\n" s.acks = {} s.out = "" if out != "": s.conn.write(out) # if this fails, the reader sees the end and we reconnect fn handle(s: mut State, r: Req, reqs: chan[Req]): match r: Publish(q, body, delay, reply): id = s.next_id s.next_id += 1 f = f"P {id} {q} {delay} {len(body)}\n" + body _cat(&s.out, f) s.pend[id] = Pending(f, reply, dummy_txt(), 0) PublishBatch(q, bodies, reply): id = s.next_id s.next_id += 1 head = f"B {id} {q} 0 {len(bodies)}" for b in bodies: head += f" {len(b)}" f = head + "\n" + "".join(bodies) _cat(&s.out, f) s.pend[id] = Pending(f, reply, dummy_txt(), 0) Subscribe(q, group, prefetch, msgs, reply): sid = s.next_id s.next_id += 1 s.subs[sid] = SubInfo(q, group, prefetch, msgs) _cat(&s.out, f"S {sid} {q} {group} {prefetch}\n") reply.send(sid) Unsubscribe(sid): if sid in s.subs: task.close(s.subs[sid].msgs) s.subs.remove(sid) _cat(&s.out, f"U {sid}\n") Ack(sid, seq, ok): key = sid * 2 tag = "A" if not ok: key += 1 tag = "N" if key in s.acks: _cat(&s.acks[key], f" {seq}") else: s.acks[key] = f"{tag} {sid} {seq}" Fetch(sid, max, wait, reply): if not (sid in s.subs): reply.send(0) else: if not (sid in s.fetches): s.fetches[sid] = [] s.fetches[sid].push(reply) if wait > 0: _cat(&s.out, f"F {sid} {max} {wait}\n") else: _cat(&s.out, f"F {sid} {max}\n") Query(line, reply): id = s.next_id s.next_id += 1 f = line.replace("{id}", str(id)) + "\n" _cat(&s.out, f) s.pend[id] = Pending(f, dummy_ok(), reply, 1) Raw(line): _cat(&s.out, line) Close: s.closed = true Reply(gen, id, ok): if gen == s.gen and id in s.pend: p = s.pend[id] s.pend.remove(id) if p.kind == 0: p.ok.send(ok) else: p.txt.send("") Json(gen, id, text): if gen == s.gen and id in s.pend: p = s.pend[id] s.pend.remove(id) p.txt.send(text) Fetched(gen, sid, n): if gen == s.gen and sid in s.fetches and len(s.fetches[sid]) > 0: s.fetches[sid][0].send(n) s.fetches[sid] = s.fetches[sid][1..] Lost(gen): if gen == s.gen: s.up = false Epoch(gen, epoch): if epoch > s.epoch: s.epoch = epoch Deliver(gen, m): if gen == s.gen and m.sid in s.subs: mm = m mm.shard = s.shard s.subs[m.sid].msgs.send(mm) fn dummy_ok() chan[bool]: c: chan[bool] = task.channel(1) return c fn dummy_txt() chan[str]: c: chan[str] = task.channel(1) return c # connect again (the next address of a failover list first, then backing off up to # 2s), then resend what's pending and resubscribe. Requests keep being taken meanwhile # (a publish waits in s.pend; a Close gives up). fn reconnect(s: mut State, reqs: chan[Req]): wait = 50 r = Req.Close s.cur = (s.cur + 1) % len(s.addrs) while true: if _dial(&s): break until = time.now_ms() + wait while not s.closed: left = until - time.now_ms() if left <= 0 or not reqs.recv_timeout(&r, left): break handle(&s, r, reqs) if s.closed: return wait = min(wait * 2, 2000) s.up = true s.gen += 1 g = s.gen cn = s.conn task.spawn(fn() reader(cn, g, reqs)) out = "" for sid, si in s.subs: out += f"S {sid} {si.queue} {si.group} {si.prefetch}\n" for id, p in s.pend: out += p.frame for sid, fs in s.fetches: for f in fs: f.send(0) s.fetches = {} s.acks = {} s.out = "" s.conn.write(out) # ---------------------------------------------------------------- reading the broker's frames fn reader(c: net.Conn, gen: int, reqs: chan[Req]): while true: line = c.read_line() if len(line) < 2 or not line.ends_with("\n"): break op = line.byte(0) pos = 1 if op == 77: # M sid seq attempt len sid = _int(line, &pos) seq = _int(line, &pos) att = _int(line, &pos) n = _int(line, &pos) body = _body(c, n) if len(body) < n: break reqs.send(Req.Deliver(gen, Msg(sid, seq, att, body, 0))) elif op == 75 or op == 69: # K id / E id msg id = _int(line, &pos) if id == 0 and op == 69: # this broker won't talk to us (unauthorized, break # or replaced by a newer primary) reqs.send(Req.Reply(gen, id, op == 75)) elif op == 86: # V epoch reqs.send(Req.Epoch(gen, _int(line, &pos))) elif op == 74: # J id len id = _int(line, &pos) n = _int(line, &pos) body = _body(c, n) if len(body) < n: break reqs.send(Req.Json(gen, id, body)) elif op == 70: # F sid n sid = _int(line, &pos) reqs.send(Req.Fetched(gen, sid, _int(line, &pos))) net.close(c) reqs.send(Req.Lost(gen)) fn _body(c: net.Conn, n: int) str: b = c.read(n) while len(b) < n: more = c.read(n - len(b)) if more == "": return b b += more return b fn _int(s: str, pos: mut int) int: n = len(s) while pos < n and s.byte(pos) == 32: pos += 1 v = 0 while pos < n: b = s.byte(pos) if b < 48 or b > 57: break v = v * 10 + (b - 48) pos += 1 return v # b += x in place (`s.field += x` would copy the whole string each time) fn _cat(b: mut str, x: str): b += x ``` ## deps/roost/check.tn ```tern # Part of the roost package (see roost.tn for the overview): input validation. # Validating input: collect every problem, then answer 422 with all of them at once. # v = roost.check() # v.required("name", u.name) # v.max_len("name", u.name, 100) # v.email("email", u.email) # v.between("age", u.age, 0, 150) # if not v.valid(): # return v.problems() # 422 {"errors": {"name": "is required", ...}} # Each field keeps its first problem. struct Check: errors: {str: str} fn check() Check: return Check({}) fn valid(v: Check) bool: return len(v.errors) == 0 fn fail(v: mut Check, field: str, message: str): if not (field in v.errors): v.errors[field] = message fn required(v: mut Check, field: str, value: str): if value.trim() == "": fail(&v, field, "is required") fn min_len(v: mut Check, field: str, value: str, n: int): if len(value) < n: fail(&v, field, f"must be at least {n} characters") fn max_len(v: mut Check, field: str, value: str, n: int): if len(value) > n: fail(&v, field, f"must be at most {n} characters") fn between(v: mut Check, field: str, value: int, lo: int, hi: int): if value < lo or value > hi: fail(&v, field, f"must be between {lo} and {hi}") fn one_of(v: mut Check, field: str, value: str, options: [str]): if not (value in options): fail(&v, field, "must be one of " + ", ".join(options)) # a plausible address: one @, something before it, a dot in the domain, no spaces fn email(v: mut Check, field: str, value: str): at = value.find("@") domain = "" if at > 0: domain = value[at + 1..] if at <= 0 or domain.find("@") >= 0 or domain.find(".") <= 0 or domain.ends_with(".") or value.contains(" "): fail(&v, field, "must be an email address") # 422 Unprocessable Content with {"errors": {field: message, ...}} fn problems(v: Check) Response: body = "{\"errors\":{" first = true for k, m in v.errors: if not first: body += "," first = false body += k.json_quote() + ":" + m.json_quote() return Response(422, "application/json", body + "}}", "", 0) ``` ## deps/roost/middleware.tn ```tern # Part of the roost package (see roost.tn for the overview): middleware. fn _fmt_us(us: int) str: if us < 1000: return str(us) + "us" if us < 1_000_000: return str(us / 1000) + "." + str(us / 100 % 10) + "ms" return str(us / 1_000_000) + "." + str(us / 100_000 % 10) + "s" fn _log_mw(req: Request, inner: fn(Request) Response) Response: t0 = http.now_us() res = inner(req) http.log(req.method + " " + req.path + " " + str(res.status) + " " + _fmt_us(http.now_us() - t0)) return res fn logger() fn(Request, fn(Request) Response) Response: return _log_mw # Rate limiting: a token bucket per client IP (client_ip), refilled at per_second tokens a # second up to burst. A request that finds its bucket empty gets 429 Too Many Requests with # Retry-After. Buckets are shared by every thread; idle ones are forgotten. fn rate_limit(per_second: float, burst: int) fn(Request, fn(Request) Response) Response: lim = http.limiter(per_second, burst) return fn(req: Request, inner: fn(Request) Response) _limited(req, inner, lim, req.client_ip()) # the same, with your own key (an API key, a user id from the session, ...) fn rate_limit_by(per_second: float, burst: int, key: fn(Request) str) fn(Request, fn(Request) Response) Response: lim = http.limiter(per_second, burst) return fn(req: Request, inner: fn(Request) Response) _limited(req, inner, lim, key(req)) fn _limited(req: Request, inner: fn(Request) Response, lim: http.Limiter, key: str) Response: wait = lim.consume(key) if wait <= 0.0: return inner(req) secs = int(wait) + 1 return Response(429, "text/plain; charset=utf-8", "too many requests", "Retry-After: " + str(secs) + "\r\n", 0) fn _list_has(list: str, item: str) bool: for x in list.split(","): if x.trim() == item: return true return false fn _cors(req: Request, inner: fn(Request) Response, origins: str) Response: origin = req.header("origin") allow = "" if origins == "*": allow = "*" elif origin != "" and _list_has(origins, origin): allow = origin if req.method == "OPTIONS" and req.header("access-control-request-method") != "": pre = Response(204, "", "", "", 0) if allow != "": asked = req.header("access-control-request-headers") pre.headers = "Access-Control-Allow-Origin: " + allow + "\r\nAccess-Control-Allow-Methods: GET, POST, PUT, PATCH, DELETE, OPTIONS\r\nAccess-Control-Max-Age: 86400\r\n" if asked != "": pre.headers += "Access-Control-Allow-Headers: " + asked + "\r\n" if allow != "*": pre.headers += "Vary: Origin\r\n" return pre res = inner(req) if allow == "*": res.headers += "Access-Control-Allow-Origin: *\r\n" elif allow != "": res.headers += "Access-Control-Allow-Origin: " + allow + "\r\nVary: Origin\r\n" return res fn cors(origins: str) fn(Request, fn(Request) Response) Response: return fn(req: Request, inner: fn(Request) Response) _cors(req, inner, origins) fn _request_id_mw(req: Request, inner: fn(Request) Response) Response: id = req.header("x-request-id") if id == "": id = http.request_id() r2 = req r2.head += "x-request-id: " + id + "\r\n" res = inner(r2) res.headers += "x-request-id: " + id + "\r\n" return res res = inner(req) res.headers += "x-request-id: " + id + "\r\n" return res fn request_id() fn(Request, fn(Request) Response) Response: return _request_id_mw fn _deliver(ch: chan[Response], inner: fn(Request) Response, req: Request): task.send(ch, inner(req)) fn _timeout(req: Request, inner: fn(Request) Response, ms: int) Response: ch: chan[Response] = task.channel(1) h = task.spawn(fn() _deliver(ch, inner, req)) res = Response(503, "text/plain; charset=utf-8", "request timed out", "", 0) left = ms while left > 0: step = min(left, 25) # (looks in on the task, so a failure answers early) if task.recv_timeout(ch, &res, step): return res left -= step if task.done(h): if task.try_recv(ch, &res): return res # the handler failed (its error is reported when the task is dropped) return Response(500, "text/plain; charset=utf-8", "internal server error", "", 0) task.cancel(h) return res fn timeout(ms: int) fn(Request, fn(Request) Response) Response: return fn(req: Request, inner: fn(Request) Response) _timeout(req, inner, ms) fn _unauthorized() Response: return Response(401, "text/plain; charset=utf-8", "unauthorized", "WWW-Authenticate: Bearer\r\n", 0) fn _bearer(req: Request, inner: fn(Request) Response, want: str) Response: if http.secure_equal(req.header("authorization"), want): return inner(req) return _unauthorized() fn bearer_auth(token: str) fn(Request, fn(Request) Response) Response: want = "Bearer " + token return fn(req: Request, inner: fn(Request) Response) _bearer(req, inner, want) fn _check(req: Request, inner: fn(Request) Response, ok: fn(Request) bool) Response: if ok(req): return inner(req) return _unauthorized() fn auth(check: fn(Request) bool) fn(Request, fn(Request) Response) Response: return fn(req: Request, inner: fn(Request) Response) _check(req, inner, check) fn _compressible(ct: str) bool: return ct.starts_with("text/") or ct.contains("json") or ct.contains("javascript") or ct.contains("xml") or ct.contains("svg") fn _gzip_mw(req: Request, inner: fn(Request) Response) Response: res = inner(req) if res.upgrade != 0 or len(res.body) < 860 or not _compressible(res.content_type): return res if not req.header("accept-encoding").contains("gzip") or res.get_header("content-encoding") != "": return res z = http.gzip(res.body) if len(z) == 0 or len(z) >= len(res.body): return res res.body = z res.headers += "Content-Encoding: gzip\r\nVary: Accept-Encoding\r\n" return res fn gzip() fn(Request, fn(Request) Response) Response: return _gzip_mw ``` ## deps/roost/response.tn ```tern # Part of the roost package (see roost.tn for the overview): responses, and the other http # functions a roost program calls by name (roost.text(...), roost.metric_add(...)); the # functions on requests and responses (req.param(...), res.set_header(...)) are http's. fn text(body: str) Response: return http.text(body) fn html(body: str) Response: return http.html(body) fn json(body: str) Response: return http.json(body) fn status(code: int, body: str) Response: return http.status(code, body) fn not_found() Response: return http.not_found() fn no_content() Response: return http.no_content() fn redirect(to: str) Response: return http.redirect(to) fn redirect_permanent(to: str) Response: return http.redirect_permanent(to) fn with_status(res: Response, code: int) Response: return http.with_status(res, code) fn file_response(path: str, req: Request) Response: return http.file_response(path, req) fn metric_add(name: str, labels: str, v: float): http.metric_add(name, labels, v) fn metric_set(name: str, labels: str, v: float): http.metric_set(name, labels, v) fn trust_proxies(n: int): http.trust_proxies(n) fn track_routes(): http.track_routes() ``` ## deps/roost/roost.tn ```tern # roost: an axum-style web framework on the standard library's http engine: routing with path # parameters, extractors, middleware, WebSockets, server-sent events and static files. # roost.Request, Response, Socket, Stream, Part and Server are http's types # (aliases), so their functions (req.param, res.set_header, ws.send, ...) are http's. # # import roost # import json # import task # # struct User: # id: int # name: str # # fn show_user(req: roost.Request) roost.Response: # id = req.param_int("id", -1) # if id < 0: # return roost.status(400, "bad id") # return roost.json(json.encode(User(id, "ada"))) # # fn bump(n: mut int) int: # n += 1 # return n # # fn main(): # hits = task.shared(0) # app state: captured by the handlers # r = roost.router() # r.get("/", fn(req) roost.text("hello")) # r.get("/users/{id}", show_user) # r.post("/hits", fn(req) roost.text(str(hits.update(bump)))) # r.layer(roost.logger()) # middleware for the routes above # roost.serve(r, 8080) # a task per connection, every core # # Handlers are closures `fn(roost.Request) roost.Response`; state they need is captured (a # `shared[T]` for state that changes). A runtime error in a handler answers 500 and the # server keeps running. Each connection runs as a task, so handlers may wait (sleep, # channels, http.get) without holding up other requests. # # Router (r.get(...) changes r in place): # roost.router() Router # r.get(path, h) r.post(path, h) r.put(path, h) r.delete(path, h) r.patch(path, h) # r.route(method, path, h) any method name; "*" matches every method # r.any(path, h) every method # r.ws(path, fn(req: roost.Request, ws: roost.Socket)) a WebSocket endpoint # r.sse(path, fn(req: roost.Request, out: roost.Stream)) a server-sent events endpoint # r.nest(prefix, sub) sub's routes under prefix ("/api" + "/users") # r.fallback(h) answers requests no route matches (default: 404) # r.post_stream(path, h) r.put_stream(path, h) the handler runs when the headers arrive and # reads the body itself (big uploads in constant memory): # req.read_body(max) str the next piece ("" at the end) req.save_body(path) int # req.next_part(&p) bool multipart: the next part's name / filename / content_type # req.read_part(max) str that part's bytes, a piece at a time req.save_part(path) int # r.max_connections(n) at most n connections at once (then accepting waits) # r.metrics(path) GET path: Prometheus metrics (requests, latency by route, ...) # roost.metric_add(name, labels, v) roost.metric_set(name, labels, v) your own counters / gauges # (labels: "" or `kind="x",zone="eu"`) # Validation: roost.check() Check; v.required / min_len / max_len / between / one_of / email / # fail(field, message); v.valid() bool; v.problems() Response (422 with every problem) # r.files(prefix, dir) static files: GET prefix/a/b.css serves dir/a/b.css (ETag, 304, # ranges, content types; ".." and dot files are refused) # roost.file_response(path, req) Response one file as a response (with the same headers; the # path is relative to the working directory, and ".." is refused) # r.layer(mw) wraps the routes added so far (and the fallback) # r.wrap(mw) wraps everything, including 404s and 405s # Paths: "/users/{id}" (one segment), "/files/{*rest}" (the rest of the path). # Static segments win over {params}, which win over {*wildcards}. A path that matches # with the wrong method gets 405 with an Allow header; GET routes also answer HEAD. # # Serving: # roost.serve(r, port) serve forever (prints the address to stderr) # roost.start(r, port) Server serve in the background; port 0 picks a free port # roost.start_on(r, host, port) Server # srv.port (a field, int) srv.stop() (closes connections, waits for their handlers) # roost.serve_tls(r, port, cert, key) HTTPS: cert and key are PEM files (the cert file # roost.start_tls(r, host, port, cert, key) Server may hold the chain); needs `import tls` # # Request fields: method, path, raw_query, head (header lines), body, params, ctx # req.param(name) str a path parameter ("" if absent); req.param_int(name, default) int # req.query(name) str from the query string, decoded; req.queries() {str: str} # req.header(name) str case-insensitive; req.headers() {str: str} (lowercase names) # req.cookie(name) str req.form(name) str (an x-www-form-urlencoded body); req.forms() {str: str} # req.parts() [roost.Part] a multipart/form-data body (file uploads): name, filename, # req.part(name) roost.Part content_type, data of each part # req.json(&value) bool decode the JSON body into value (false: left unchanged) # req.params() {str: str} every path parameter # req.add_header(name, value) (middleware) add a request header for inner handlers # req.set_ctx(key, value) req.ctx_value(key) str values middleware passes inward # req.peer() str "1.2.3.4:5678", the connection's address # req.client_ip() str the client's IP (see roost.trust_proxies(n) behind a proxy) # req.matched_route() str the matched route's pattern ("/users/{id}"), once # roost.track_routes() was called ("" before, or if none matched) # # Responses (fields: status, content_type, body, headers, upgrade): # roost.text(s) roost.html(s) roost.json(s) roost.status(code, body) roost.not_found() # roost.redirect(url) (302) roost.redirect_permanent(url) (308) roost.no_content() # res.set_header(name, value) res.set_cookie(name, value) res.set_cookie_with(name, value, attrs) # res.get_header(name) str roost.with_status(res, code) Response # # Middleware is a closure fn(req: roost.Request, next: fn(roost.Request) roost.Response) roost.Response: # roost.logger() "GET /path 200 1.2ms" per request, to stderr (buffered) # roost.rate_limit(per_second, burst) 429 + Retry-After past a token bucket per client IP # roost.rate_limit_by(per_second, burst, key) ... per key(req) (fn(roost.Request) str) # roost.cors(origins) "*" or "https://a.com, https://b.com"; answers preflights # roost.request_id() x-request-id on the request (if missing) and the response # roost.timeout(ms) runs the handler in a task; 503 when it takes too long # roost.bearer_auth(token) 401 unless "Authorization: Bearer " # roost.auth(check) 401 unless check(req) (check: fn(roost.Request) bool) # roost.gzip() compresses text/JSON bodies for clients that accept gzip # # WebSockets (roost.Socket): # ws.next(&msg) bool the next text or binary message; false once closed # ws.recv() str the next message ("" once closed) # ws.is_binary() bool whether the last message received was binary # ws.send(text) bool ws.send_binary(data) bool ws.ping(data) bool (false: gone) # ws.close() ws.is_open() bool # (a WebSocket client is http.ws_connect(url)) # Pings are answered automatically; fragmented messages are put together. # # Server-sent events (roost.Stream): # out.send(data) bool one event (multi-line data is fine); false: client gone # out.event(name, data) bool a named event out.comment(text) bool (keep-alive) # out.forward(ch) bool every message from a chan[str] as an event, until ch closes # (false: client gone, noticed within 1s); waiting messages share a write # # HTTP client: http.get, http.post, http.fetch(http.request(method, url)) (see `tern guide http`); # roost has none of its own, as axum leaves the client to reqwest. # # Environment: TERN_ASYNC_THREADS (worker threads), TERN_WEB_IDLE_SEC (60), # TERN_WEB_HEADER_SEC (10), TERN_WEB_MAX_BODY_MB (16), TERN_WEB_WS_MAX_MB (16). # Not available on wasm (no sockets or threads there). import http import task type Request = http.Request type Response = http.Response type Socket = http.Socket type Stream = http.Stream type Part = http.Part type Server = http.Server struct Route: method: str pattern: str handler: fn(Request) Response ws: fn(Request, Socket) sse: fn(Request, Stream) layers: int # how many of the router's layers existed when it was added stream: bool # the handler reads the body itself (post_stream / put_stream) struct Router: routes: [Route] layers: [fn(Request, fn(Request) Response) Response] wraps: [fn(Request, fn(Request) Response) Response] fallback_handler: fn(Request) Response fallback_layers: int has_fallback: bool max_conns: int # 0: no limit (see max_connections) ``` ## deps/roost/router.tn ```tern # Part of the roost package (see roost.tn for the overview): the router. fn _not_found_handler(req: Request) Response: return not_found() fn router() Router: return Router([], [], [], _not_found_handler, 0, false, 0) # GET path answers with Prometheus metrics (text format 0.0.4): tern_http_requests_total and # tern_http_request_duration_seconds by route, method and status (counted from now on, for # every server in the program), tern_http_connections, process CPU / memory / uptime, and the # program's own roost.metric_add / roost.metric_set values. fn metrics(r: mut Router, path: str): http.enable_metrics() route(&r, "GET", path, _metrics_page) fn _metrics_page(req: Request) Response: return Response(200, "text/plain; version=0.0.4; charset=utf-8", http.metrics_text(), "", 0) # At most n connections at once (0: no limit, the default; TERN_WEB_MAX_CONNS sets it too). # Past the limit the server stops accepting until a connection closes; new clients wait in # the kernel's queue rather than being refused. fn max_connections(r: mut Router, n: int): r.max_conns = n fn route(r: mut Router, method: str, path: str, h: fn(Request) Response): r.routes.push(Route(method.upper(), path, h, http.no_ws, http.no_sse, len(r.layers), false)) # r.get(path, h), r.post, r.put, r.delete, r.patch fn get(r: mut Router, path: str, h: fn(Request) Response): route(&r, "GET", path, h) fn post(r: mut Router, path: str, h: fn(Request) Response): route(&r, "POST", path, h) fn put(r: mut Router, path: str, h: fn(Request) Response): route(&r, "PUT", path, h) fn delete(r: mut Router, path: str, h: fn(Request) Response): route(&r, "DELETE", path, h) fn patch(r: mut Router, path: str, h: fn(Request) Response): route(&r, "PATCH", path, h) fn any(r: mut Router, path: str, h: fn(Request) Response): route(&r, "*", path, h) # Streaming uploads: the handler runs as soon as the headers arrive, and reads the body # itself, a piece at a time (no size limit, constant memory). req.body is "". fn post_stream(r: mut Router, path: str, h: fn(Request) Response): r.routes.push(Route("POST", path, h, http.no_ws, http.no_sse, len(r.layers), true)) fn put_stream(r: mut Router, path: str, h: fn(Request) Response): r.routes.push(Route("PUT", path, h, http.no_ws, http.no_sse, len(r.layers), true)) # r.files("/assets", "./public"): GET /assets/app.js serves ./public/app.js (a directory # serves its index.html). Content types from the file extension, ETag / Last-Modified with # 304 answers, single byte ranges (206), and big files streamed from disk. Paths with ".." # or dot files (other than .well-known) are 404. fn files(r: mut Router, prefix: str, dir: str): p = prefix while p.ends_with("/"): p = p[..len(p) - 1] h = fn(req: Request) http.static_file(dir, req.param("path"), req) route(&r, "GET", p + "/{*path}", h) if p != "": route(&r, "GET", p, h) else: route(&r, "GET", "/", h) fn ws(r: mut Router, path: str, h: fn(Request, Socket)): r.routes.push(Route("GET", path, http.ws_upgrade, h, http.no_sse, len(r.layers), false)) fn sse(r: mut Router, path: str, h: fn(Request, Stream)): r.routes.push(Route("GET", path, http.sse_open, http.no_ws, h, len(r.layers), false)) fn fallback(r: mut Router, h: fn(Request) Response): r.fallback_handler = h r.fallback_layers = len(r.layers) r.has_fallback = true fn layer(r: mut Router, mw: fn(Request, fn(Request) Response) Response): r.layers.push(copy(mw)) fn wrap(r: mut Router, mw: fn(Request, fn(Request) Response) Response): r.wraps.push(copy(mw)) # h inside mws[first], mws[first + 1], ... (the last one is outermost) fn _compose(h: fn(Request) Response, mws: [fn(Request, fn(Request) Response) Response], first: int) fn(Request) Response: out = h i = first while i < len(mws): mw = mws[i] inner = out out = fn(req: Request) mw(req, inner) i += 1 return out fn _join_path(prefix: str, pattern: str) str: p = prefix if p.ends_with("/"): p = p[..len(p) - 1] if pattern == "/": if p == "": return "/" return p return p + pattern fn nest(r: mut Router, prefix: str, sub: Router): for rt in sub.routes: h = _compose(_compose(rt.handler, sub.layers, rt.layers), sub.wraps, 0) r.routes.push(Route(rt.method, _join_path(prefix, rt.pattern), h, rt.ws, rt.sse, len(r.layers), rt.stream)) if sub.has_fallback: fb = _compose(_compose(sub.fallback_handler, sub.layers, sub.fallback_layers), sub.wraps, 0) r.routes.push(Route("*", _join_path(prefix, "/{*rest}"), fb, http.no_ws, http.no_sse, len(r.layers), false)) fn _405(allow: str) Response: return Response(405, "text/plain; charset=utf-8", "method not allowed", "Allow: " + allow + "\r\n", 0) fn _method_not_allowed(allow: str) fn(Request) Response: return fn(req: Request) _405(allow) # the routes with their middleware applied, plus a 405 answer for each path fn _finished_routes(r: Router) [Route]: out: [Route] = [] allow: {str: str} = {} first: {str: int} = {} has_any: {str: bool} = {} for rt in r.routes: h = _compose(_compose(rt.handler, r.layers, rt.layers), r.wraps, 0) out.push(Route(rt.method, rt.pattern, h, rt.ws, rt.sse, rt.layers, rt.stream)) if rt.method == "*": has_any[rt.pattern] = true continue m = rt.method if m == "GET": m = "GET, HEAD" if rt.pattern in allow: allow[rt.pattern] = allow[rt.pattern] + ", " + m first[rt.pattern] = min(first[rt.pattern], rt.layers) else: allow[rt.pattern] = m first[rt.pattern] = rt.layers for pat, methods in allow: if not (pat in has_any): h = _compose(_compose(_method_not_allowed(methods), r.layers, first[pat]), r.wraps, 0) out.push(Route("*", pat, h, http.no_ws, http.no_sse, 0, false)) return out ``` ## deps/roost/serve.tn ```tern # Part of the roost package (see roost.tn for the overview): serving, on http's engine. fn _start(r: Router, host: str, port: int, cert: str, key: str) Server: s = http.new_server() for rt in _finished_routes(r): s.add_route(rt.method, rt.pattern, rt.handler, rt.ws, rt.sse, rt.stream) fb = _compose(_compose(r.fallback_handler, r.layers, r.fallback_layers), r.wraps, 0) if r.max_conns > 0: s.limit_connections(r.max_conns) if cert != "": s.listen_tls(host, port, fb, cert, key) else: s.listen(host, port, fb) return s fn start_on(r: Router, host: str, port: int) Server: return _start(r, host, port, "", "") fn start_tls(r: Router, host: str, port: int, cert: str, key: str) Server: return _start(r, host, port, cert, key) fn serve_tls(r: Router, port: int, cert: str, key: str): _start(r, "", port, cert, key).wait() fn start(r: Router, port: int) Server: return start_on(r, "", port) fn serve(r: Router, port: int): start_on(r, "", port).wait() fn stop(s: Server): s.stop() ``` ## deps/sentry/sentry.tn ```tern # sentry: report errors and messages to Sentry (sentry.io or self-hosted) from any task. # # import sentry # import tls # for an https:// DSN # # o = sentry.options() # defaults; change the fields you need # o.release = "shop@1.4.2" # c = sentry.init(os.env("SENTRY_DSN", ""), o) # an empty DSN: capturing does nothing # sentry.install(c) # runtime errors (crashes, failed tasks, 500s) # r.layer(sentry.middleware(c)) # ...with the request and route they happened in # c.capture_message("disk almost full", "warning") # c.capture_result(fs.try_read(path)) # an Err becomes an error event # c.close() # before main returns: send what's queued # # sentry.options() Options release ($SENTRY_RELEASE), environment ($SENTRY_ENVIRONMENT # or "production"), server_name ($HOSTNAME), sample_rate 1.0, # max_queue 100, breadcrumbs 50, timeout_ms 10000, scrub [...] # sentry.init(dsn str, o Options) Client # c.ok() bool false for an empty or bad DSN; c.error() says which # c.capture_message(msg str, level str) str "fatal" "error" "warning" "info" "debug" # c.capture_error(r os.ErrorReport) str a runtime error, with its native stack trace # c.capture_result(r Result[T, str]) str Err(e): an error event with message e; Ok: none # (each returns the event id, "" if nothing was queued) # c.flush(timeout_ms int) bool wait until everything queued was sent (or dropped) # c.close() flush for up to 2 s, then stop sending # c.sent() int c.dropped() int events delivered / dropped (queue full, rate limited, failed) # sentry.install(c) os.on_error reports every runtime error to c, sending each # before the handler returns (so one that ends the program # is sent too) # sentry.middleware(c) web middleware: events from a request carry its method, # URL, headers, and the route as the transaction name # sentry.set_tag(key, value) sentry.set_extra(key, value) sentry.set_user(id, email) # sentry.clear_scope() data for events from this task (and tasks it spawns) # # Events are queued (up to max_queue) and sent by a background task, so capturing never waits # for the network. Sending retries after network errors and 5xx (after 1 s, 2 s, 4 s), and # follows Sentry's rate limits (429, X-Sentry-Rate-Limits): events are dropped until the # time passes. Header, query parameter, and field names containing one of o.scrub (lowercase: # auth, cookie, password, passwd, secret, token, key, session, credential; add your own) have # their values replaced with "[Filtered]". Each event carries the last o.breadcrumbs records # from log.recent (warn and up; log.keep("info") keeps more). Stack traces are native # addresses that Sentry symbolicates from the binary's debug file (docs/packages/sentry.md). import crypto import fs import http import json import log import os import task import time const SDK_VERSION = "0.1.0" const FILTERED = "[Filtered]" struct Options: release: str environment: str server_name: str sample_rate: float # the share of events sent, 0.0 to 1.0 max_queue: int # events waiting to be sent; more are dropped breadcrumbs: int # recent log records sent with each event timeout_ms: int # per request to Sentry scrub: [str] # names to hide (lowercase; a name containing one is hidden) fn options() Options: return Options(os.env("SENTRY_RELEASE", ""), os.env("SENTRY_ENVIRONMENT", "production"), os.env("HOSTNAME", ""), 1.0, 100, 50, 10000, ["auth", "cookie", "password", "passwd", "secret", "token", "key", "session", "credential"]) struct Stats: pending: int # queued or being sent sent: int dropped: int hold_until: float # rate limited until this time.now() struct Client: dsn: str endpoint: str # :///api//envelope/ ("" when off) auth: str # the X-Sentry-Auth header why: str opts: Options queue: chan[str] # envelopes for the sender stats: shared[Stats] fn init(dsn: str, o: Options) Client: q: chan[str] = task.channel(0) c = Client(dsn, "", "", "", o, q, task.shared(Stats(0, 0, 0, 0.0))) if dsn == "": c.why = "sentry: no DSN, so nothing is sent" return c d = parse_dsn(dsn) if d.why != "": c.why = "sentry: bad DSN: " + d.why return c c.endpoint = d.endpoint c.auth = d.auth task.spawn(fn() _send_loop(c)) return c fn ok(c: Client) bool: return c.endpoint != "" fn error(c: Client) str: return c.why fn sent(c: Client) int: return c.stats.get().sent fn dropped(c: Client) int: return c.stats.get().dropped # ---- DSN: ://[:]@[:][/]/ struct Dsn: endpoint: str auth: str why: str fn parse_dsn(dsn: str) Dsn: scheme = "" for s in ["https://", "http://"]: if dsn.starts_with(s): scheme = s if scheme == "": return Dsn("", "", "it must start with https:// or http://") rest = dsn[len(scheme)..] at = rest.find("@") if at <= 0: return Dsn("", "", "no public key before @") key = rest[..at] colon = key.find(":") if colon >= 0: key = key[..colon] hostpath = rest[at + 1..] while hostpath.ends_with("/"): hostpath = hostpath[..len(hostpath) - 1] last = len(hostpath) - 1 while last >= 0 and hostpath[last] != "/": last -= 1 if last <= 0: return Dsn("", "", "no project id at the end") project = hostpath[last + 1..] if project == "" or project.to_int(-1) < 0: return Dsn("", "", "the project id isn't a number") endpoint = scheme + hostpath[..last] + "/api/" + project + "/envelope/" auth = f"Sentry sentry_version=7, sentry_key={key}, sentry_client=sentry.tern/{SDK_VERSION}" return Dsn(endpoint, auth, "") # ---- the task's scope: tags, extra, user, and the request (kept in a task-local) struct User: id: str email: str struct RequestInfo: method: str url: str query_string: str headers: {str: str} env: {str: str} struct Scope: tags: {str: str} extra: {str: str} user: User request: RequestInfo transaction: str fn _empty_scope() Scope: return Scope({}, {}, User("", ""), RequestInfo("", "", "", {}, {}), "") fn _scope() Scope: s = _empty_scope() text = task.local_get("sentry.scope") if text != "": json.decode(text, &s) return s fn _save(s: Scope): task.local_set("sentry.scope", json.encode(s)) fn set_tag(key: str, value: str): s = _scope() s.tags[key] = value _save(s) fn set_extra(key: str, value: str): s = _scope() s.extra[key] = value _save(s) fn set_user(id: str, email: str): s = _scope() s.user = User(id, email) _save(s) fn clear_scope(): task.local_set("sentry.scope", "") # ---- scrubbing fn _secret(name: str, scrub: [str]) bool: n = name.lower() for s in scrub: if s != "" and n.contains(s): return true return false fn _scrub_map(m: {str: str}, scrub: [str]) {str: str}: out: {str: str} = {} for k, v in m: out[k] = FILTERED if _secret(k, scrub) else v return out fn _scrub_query(q: str, scrub: [str]) str: if q == "": return q parts: [str] = [] for p in q.split("&"): eq = p.find("=") name = p[..eq] if eq >= 0 else p if eq >= 0 and _secret(url_decode(name), scrub): parts.push(name + "=" + FILTERED) else: parts.push(p) return "&".join(parts) # ---- capturing fn capture_message(c: Client, message: str, level: str) str: return _capture(c, level, "\"message\":{\"formatted\":" + json_quote(message) + "},", _scope()) fn capture_result[T](c: Client, r: Result[T, str]) str: if r is Err(e): return capture_message(c, e, "error") return "" # (with the scope of the task that failed: the report carries its task-locals) fn capture_error(c: Client, r: os.ErrorReport) str: s = _empty_scope() if "sentry.scope" in r.locals and r.locals["sentry.scope"] != "": json.decode(r.locals["sentry.scope"], &s) return _capture(c, "error", _exception(r) + _debug_meta(r), s) # Queues an event (the level and the JSON fields that make it what it is); its id or "". fn _capture(c: Client, level: str, body: str, scope: Scope) str: if not c.ok() or not _sampled(c.opts.sample_rate) or not _admit(c): return "" id = crypto.uuid4().replace("-", "") now = time.now() ev = "{\"event_id\":\"" + id + "\",\"timestamp\":" + json_quote(time.rfc3339(now)) ev += ",\"platform\":\"native\",\"level\":" + json_quote(_level(level)) + ",\"logger\":\"tern\"," ev += body + _common(c.opts) + _scope_json(scope, c.opts.scrub) + _breadcrumbs(c.opts) + "}" head = "{\"event_id\":\"" + id + "\",\"sent_at\":" + json_quote(time.rfc3339(now)) + ",\"dsn\":" + json_quote(c.dsn) + "}" item = "{\"type\":\"event\",\"length\":" + str(len(ev)) + "}" c.queue.send(head + "\n" + item + "\n" + ev + "\n") return id fn _level(l: str) str: if l == "warn": return "warning" if l in ["fatal", "error", "warning", "info", "debug"]: return l return "error" fn _sampled(rate: float) bool: if rate >= 1.0: return true return float(crypto.random_int(0, 999999)) < rate * 1000000.0 # room in the queue, and not rate limited: count it as pending fn _admit(c: Client) bool: max = c.opts.max_queue now = time.now() fn admit(s: mut Stats) bool: if s.pending >= max or s.hold_until > now: s.dropped += 1 return false s.pending += 1 return true return c.stats.update(admit) fn _common(o: Options) str: out = "\"sdk\":{\"name\":\"sentry.tern\",\"version\":\"" + SDK_VERSION + "\"}" out += ",\"contexts\":{\"runtime\":{\"name\":\"tern\"}}" if o.release != "": out += ",\"release\":" + json_quote(o.release) if o.environment != "": out += ",\"environment\":" + json_quote(o.environment) if o.server_name != "": out += ",\"server_name\":" + json_quote(o.server_name) return out fn _scope_json(s: Scope, scrub: [str]) str: out = "" if len(s.tags) > 0: out += ",\"tags\":" + json.encode(s.tags) if len(s.extra) > 0: out += ",\"extra\":" + json.encode(_scrub_map(s.extra, scrub)) if s.user.id != "" or s.user.email != "": out += ",\"user\":{\"id\":" + json_quote(s.user.id) + ",\"email\":" + json_quote(s.user.email) + "}" if s.request.method != "": r = s.request r.headers = _scrub_map(r.headers, scrub) r.query_string = _scrub_query(r.query_string, scrub) out += ",\"request\":" + json.encode(r) if s.transaction != "": out += ",\"transaction\":" + json_quote(s.transaction) return out struct Crumb: timestamp: str level: str category: str message: str data: {str: str} fn _breadcrumbs(o: Options) str: if o.breadcrumbs <= 0: return "" crumbs: [Crumb] = [] for r in log.recent(o.breadcrumbs): crumbs.push(Crumb(time.rfc3339(r.time), _level(r.level), "log", r.msg, _scrub_map(r.fields, o.scrub))) if len(crumbs) == 0: return "" return ",\"breadcrumbs\":{\"values\":" + json.encode(crumbs) + "}" # ---- runtime errors: an exception with a native stack trace, and the image to symbolicate it fn _hex(n: int) str: if n == 0: return "0x0" digits = "0123456789abcdef" out = "" v = n while v != 0: out = digits[v & 15] + out v = v >>> 4 return "0x" + out fn _in_app(f: str) bool: if f == "" or f.starts_with("tern_") or f.starts_with("_") or f.starts_with("<"): return f.starts_with(" 0: return not (f[..dot] in ["roost", "web", "pg", "redis", "auth", "html", "task", "os", "fs", "json", "log", "net", "tls", "crypto", "time", "sqlite", "http", "sentry", "queue"]) return not (f in ["start", "thread_start", "_pthread_start", "__libc_start_main", "__libc_start_call_main", "clone", "clone3"]) fn _exception(r: os.ErrorReport) str: kind = "runtime error" value = r.message i = r.message.find("): ") if r.message.starts_with("runtime error") and i > 0: p = r.message.find(" (") kind = r.message[..p] value = r.message[i + 3..] # frames: outermost first (Sentry's order), without the reporter's own fs_json: [str] = [] n = len(r.frames) skip = 0 while skip < n and (r.frames[skip].function.starts_with("tern_report") or r.frames[skip].function == "tern_panic"): skip += 1 top = skip # the frame the file and line belong to: the failing function's for j in skip..n: if r.frames[j].function == r.function or _in_app(r.frames[j].function): top = j break k = n - 1 while k >= skip: f = r.frames[k] one = "{\"instruction_addr\":\"" + _hex(f.pc) + "\",\"in_app\":" + str(_in_app(f.function)) if f.function != "": one += ",\"function\":" + json_quote(f.function) if k == top and r.file != "": one += ",\"filename\":" + json_quote(r.file) + ",\"lineno\":" + str(r.line) fs_json.push(one + "}") k -= 1 st = "" if len(fs_json) > 0: st = ",\"stacktrace\":{\"frames\":[" + ",".join(fs_json) + "]}" return "\"exception\":{\"values\":[{\"type\":" + json_quote(kind) + ",\"value\":" + json_quote(value) + st + "}]}," # the executable, for Sentry to find its debug file: Mach-O by UUID, ELF by GNU build id fn _debug_meta(r: os.ErrorReport) str: if r.build_id == "" or r.image_base == 0: return "" id = (r.build_id + "0".repeat(32))[..32] elf = fs.exists("/proc/self/exe") if elf: # the build id's first three fields, little-endian (a GUID) id = id[6..8] + id[4..6] + id[2..4] + id[..2] + id[10..12] + id[8..10] + id[14..16] + id[12..14] + id[16..] uuid = id[..8] + "-" + id[8..12] + "-" + id[12..16] + "-" + id[16..20] + "-" + id[20..32] img = "{\"type\":\"" + ("elf" if elf else "macho") + "\",\"debug_id\":\"" + uuid + "\",\"image_addr\":\"" + _hex(r.image_base) + "\"" if elf: img += ",\"code_id\":\"" + r.build_id + "\"" return "\"debug_meta\":{\"images\":[" + img + ",\"code_file\":" + json_quote(os.args()[0]) + "}]}," # ---- sending fn _send_loop(c: Client): env = "" while c.queue.next(&env): _deliver(c, env) fn _deliver(c: Client, env: str): headers = http.header_line("Content-Type", "application/x-sentry-envelope") + http.header_line("X-Sentry-Auth", c.auth) delays = [1000, 2000, 4000] sent = false for attempt in 0..4: res = http.fetch(http.ClientRequest("POST", c.endpoint, headers, env, c.opts.timeout_ms, 0)) hold = _retry_after(res) if hold > 0.0: _hold(c, hold) if res.status >= 200 and res.status < 300: sent = true break if res.status != 0 and res.status < 500: break # 429, or a 4xx that won't change on retrying if attempt < 3: task.sleep(delays[attempt]) fn done(s: mut Stats): s.pending -= 1 if sent: s.sent += 1 else: s.dropped += 1 c.stats.update(done) # how long Sentry asks us to stop sending error events (seconds; 0: it doesn't) fn _retry_after(res: http.Response) float: h = res.get_header("x-sentry-rate-limits") if h != "": best = 0.0 for entry in h.split(","): parts = entry.trim().split(":") secs = parts[0].to_float(0.0) cats = parts[1] if len(parts) > 1 else "" applies = cats == "" for cat in cats.split(";"): if cat == "error" or cat == "default": applies = true if applies and secs > best: best = secs return best if res.status == 429: return res.get_header("retry-after").to_float(60.0) return 0.0 fn _hold(c: Client, secs: float): until = time.now() + secs fn set(s: mut Stats): if until > s.hold_until: s.hold_until = until c.stats.update(set) fn flush(c: Client, timeout_ms: int) bool: deadline = time.mono_ms() + timeout_ms while c.stats.get().pending > 0: if time.mono_ms() >= deadline: return false task.sleep(10) return true fn close(c: Client): flush(c, 2000) c.queue.close() # ---- integrations fn install(c: Client): os.on_error(fn(r) _report(c, r)) # a runtime error: queue it and wait until it's sent (the program may be ending) fn _report(c: Client, r: os.ErrorReport): if capture_error(c, r) != "": flush(c, c.opts.timeout_ms) fn middleware(c: Client) fn(http.Request, fn(http.Request) http.Response) http.Response: http.track_routes() scrub = c.opts.scrub return fn(req: http.Request, next: fn(http.Request) http.Response) _around(req, next, scrub) fn _around(req: http.Request, next: fn(http.Request) http.Response, scrub: [str]) http.Response: s = _scope() host = req.header("host") proto = req.header("x-forwarded-proto") if proto == "": proto = "http" env: {str: str} = {"REMOTE_ADDR": req.client_ip()} s.request = RequestInfo(req.method, proto + "://" + host + req.path, req.raw_query, req.headers(), env) route = req.matched_route() s.transaction = req.method + " " + (route if route != "" else req.path) _save(s) res = next(req) clear_scope() return res ``` ## deps/sqlgen/emit.tn ```tern # sqlgen: from described queries to Tern code (see sqlgen.tn) enum Database: Postgres(p: pg.Pool) Sqlite(d: sqlite.Db) struct Col: name: str ty: str # the Tern type: int, float, bool, str, or Option of one db_type: str # as the database calls it table: str # the table a column comes from ("" for an expression, a parameter) enum_oid: int # a PostgreSQL enum type (0: not one): ty is the Tern enum made for it struct Described: q: Query sql: str params: [Col] cols: [Col] model: str # the table whose struct the rows use ("": a struct of its own) fn reserved() [str]: # builtins a function can't be named return ["print", "len", "push", "pop", "chr", "copy", "float", "int", "sqrt", "abs", "min", "max", "str", "join", "split", "find", "contains", "starts_with", "ends_with", "replace", "trim", "upper", "lower", "to_int", "to_float", "byte", "repeat", "get", "remove", "keys", "values", "is_null", "string_at", "map", "filter", "sort_by", "url_decode", "json_quote", "panic", "assert", "main"] fn keywords() [str]: return ["struct", "fn", "if", "elif", "else", "while", "for", "in", "return", "parallel", "do", "mut", "take", "true", "false", "and", "or", "not", "break", "continue", "pass", "import", "from", "extern", "export", "null", "enum", "const", "match", "is"] # a PostgreSQL type name -> the Tern type its values are read as (text for the rest: # timestamps, uuid, json, arrays, enums, ...) fn pg_tern_type(pg_type: str) str: if pg_type.starts_with("_") and not is_time(pg_type[1..]) and pg_tern_type(pg_type[1..]) != "str": return "[" + pg_tern_type(pg_type[1..]) + "]" # int8[] -> [int] if pg_type in ["_text", "_varchar", "_bpchar", "_name", "_uuid", "_citext"]: return "[str]" if is_time(pg_type): return "float" # seconds since 1970, like time.now() if pg_type in ["int2", "int4", "int8", "oid"]: return "int" if pg_type in ["float4", "float8", "numeric"]: return "float" if pg_type == "bool": return "bool" return "str" fn describe_query(dbase: Database, q: Query, cols_of: {str: str}) Result[Described, str]: where = f"{q.file}:{q.line}: {q.name}" match dbase: Postgres(p): pr = prepare(q, "$")? if len(pr.col_types) > 0: return Err(f"{where}: PostgreSQL gives the column types; to read one differently, cast it in the SQL (price::text AS price)") s = p.describe(pr.sql).map_err(fn(e) where + ": " + e.replace("pg: describe failed: ", ""))? params: [Col] = [] for i, t in s.params: name = pr.params[i] if i < len(pr.params) else f"arg{i + 1}" eo = s.param_oids[i] if s.param_kinds[i] == "e" else 0 ty = enum_name(t) if eo != 0 else pg_tern_type(t) if name in pr.param_types: ty = pr.param_types[name] eo = 0 if name in pr.nullable: ty = f"Option[{ty}]" params.push(Col(name, ty, t, "", eo)) raw: [Col] = [] nn: [bool] = [] for c in s.columns: t = c.base_type if c.kind == "d" else c.type_name raw.push(Col(c.name, enum_name(t) if c.kind == "e" else pg_tern_type(t), t, c.table, c.type_oid if c.kind == "e" else 0)) nn.push(c.not_null) return _finish(q, pr, params, raw, nn) Sqlite(d): pr = prepare(q, "?")? s = d.describe(pr.sql).map_err(fn(e) where + ": " + e.replace("sqlite: describe failed: ", ""))? guessed = infer_params(pr.sql, len(s.params), cols_of) params: [Col] = [] for i, pn in s.params: name = f"arg{i + 1}" if i < len(pr.params): name = pr.params[i] elif pn != "" and not pn.starts_with("?"): name = pn[1..] ty = guessed[i] if guessed[i] != "" else "str" if name in pr.param_types: ty = pr.param_types[name] if name in pr.nullable: ty = f"Option[{ty}]" params.push(Col(name, ty, "", "", 0)) raw: [Col] = [] nn: [bool] = [] for c in s.columns: ty = sqlite_type(c.decl_type) if c.name in pr.col_types: ty = pr.col_types[c.name] raw.push(Col(c.name, ty, c.decl_type, c.table, 0)) nn.push(c.not_null) return _finish(q, pr, params, raw, nn) # names checked, nullability decided: a column is an Option unless it's a NOT NULL table # column in a query without outer joins, or the query says otherwise ("col!" / "col?") fn _finish(q: Query, pr: Prepared, params: [Col], raw: [Col], not_null: [bool]) Result[Described, str]: where = f"{q.file}:{q.line}: {q.name}" for p in params: if not is_ident(p.name) or p.name in keywords() or p.name in ["pool", "conn"]: return Err(f"{where}: `{p.name}` can't be a parameter name; use @name parameters") outer = has_outer_join(pr.sql) cols: [Col] = [] names: {str: bool} = {} for i, c in raw: if not is_ident(c.name) or c.name in keywords(): return Err(f"{where}: column `{c.name}` can't be a field name; name it with AS, like `count(*) AS n`") if c.name in names: return Err(f"{where}: two columns are named {c.name}; rename one with AS") names[c.name] = true nullable = not not_null[i] or outer if c.name in pr.forced: nullable = pr.forced[c.name] cols.push(Col(c.name, f"Option[{c.ty}]" if nullable else c.ty, c.db_type, "" if outer or c.name in pr.forced or c.name in pr.col_types else c.table, c.enum_oid)) for f in pr.forced.keys(): if not (f in names): return Err(f"{where}: \"{f}!\" / \"{f}?\" doesn't name a column it returns") if q.cmd != "exec" and len(cols) == 0: return Err(f"{where}: returns no rows; use :exec") return Ok(Described(q, pr.sql, params, cols, "")) # LEFT / RIGHT / FULL joins can make any table's columns null fn has_outer_join(sql: str) bool: words = sql.upper().replace("\n", " ").replace("\t", " ").split(" ").filter(fn(w) w != "") for i in 0..len(words) - 1: if words[i] in ["LEFT", "RIGHT", "FULL"] and words[i + 1] in ["JOIN", "OUTER"]: return true return false fn emit(dbase: Database, ds: [Described], enums: [EnumType], sources: str) str: lite = dbase is Sqlite(_) module = "sqlite" if lite else "pg" out = f"# Code generated by sqlgen from {sources}. DO NOT EDIT.\n\nimport {module}\n" for e in enums: out += emit_enum(e) done: {str: bool} = {} for d in ds: # the tables' structs first if d.model != "" and not (d.model in done): done[d.model] = true out += f"\n# a row of {d.cols[0].table}\nstruct {d.model}:\n" for c in d.cols: out += f" {c.name}: {c.ty}\n" helpers: {str: Col} = {} # _text_or_null_ helpers, by name for d in ds: out += "\n" + emit_query(dbase, d) for c in d.params: if c.ty.starts_with("Option["): helpers[helper_suffix(c)] = Col("v", c.ty[7..len(c.ty) - 1], c.db_type, "", c.enum_oid) for k in helpers.keys().sort_by(fn(k) k): v = helpers[k] out += f"\nfn _text_or_null_{k}(x: Option[{v.ty}]) str:\n if x is Some(v):\n return {to_text(v, lite)}\n return {module}.NULL\n" if out.contains("time.format("): out = out.replace(f"\nimport {module}\n", f"\nimport {module}\nimport time\n") return out fn emit_query(dbase: Database, d: Described) str: q = d.q lite = dbase is Sqlite(_) single = q.cmd != "exec" and len(d.cols) == 1 row = camel(q.name) + "Row" if single: row = "_" + row if d.model != "": row = d.model out = "" if q.cmd != "exec" and d.model == "": out += f"struct {row}:\n" for c in d.cols: out += f" {c.name}: {c.ty}\n" out += "\n" args = ["conn: sqlite.Db" if lite else "pool: pg.Pool"] vals: [str] = [] for c in d.params: args.push(f"{c.name}: {c.ty}") vals.push(to_text(c, lite)) sql = quote(d.sql) ps = "[" + ", ".join(vals) + "]" item = d.cols[0].ty if single else row pick = f".{d.cols[0].name}" if single else "" for line in q.docs: out += ("# " + line).trim() + "\n" head = f"fn {q.name}(" + ", ".join(args) + ")" if lite: if q.cmd == "exec": return out + f"{head} int:\n return conn.exec_count({sql}, {ps})\n" body = f" _rows: [{row}] = []\n conn.query_into({sql}, {ps}, &_rows)\n" if q.cmd == "one": return out + f"{head} Option[{item}]:\n{body} if len(_rows) == 0:\n return None\n return Some(_rows[0]{pick})\n" if single: return out + f"{head} [{item}]:\n{body} return _rows.map(fn(_x) _x{pick})\n" return out + f"{head} [{item}]:\n{body} return _rows\n" if q.cmd == "exec": return out + f"{head} Result[int, str]:\n return pool.run({sql}, {ps})\n" if q.cmd == "one" and single: return out + f"{head} Result[Option[{item}], str]:\n _r = pool.fetch_one[{row}]({sql}, {ps})?\n if _r is Some(_row):\n return Ok(Some(_row{pick}))\n return Ok(Option[{item}].None)\n" if q.cmd == "one": return out + f"{head} Result[Option[{item}], str]:\n return pool.fetch_one[{row}]({sql}, {ps})\n" if single: return out + f"{head} Result[[{item}], str]:\n _rows = pool.fetch[{row}]({sql}, {ps})?\n return Ok(_rows.map(fn(_x) _x{pick}))\n" return out + f"{head} Result[[{item}], str]:\n return pool.fetch[{row}]({sql}, {ps})\n" # a parameter as the text the database reads (SQLite keeps booleans as 1 / 0) fn to_text(c: Col, lite: bool) str: if c.ty.starts_with("Option["): return f"_text_or_null_{helper_suffix(c)}({c.name})" if c.enum_oid != 0: # its label return f"{label_fn(c.db_type)}({c.name})" if c.ty == "float" and is_time(c.db_type): # microseconds, UTC return f"time.format({c.name}, \"%Y-%m-%d %H:%M:%S.%f+00\")" if c.ty.starts_with("["): return f"pg.array_{c.ty[1..len(c.ty) - 1]}({c.name})" if c.ty == "str": return c.name if c.ty == "bool": return f"\"1\" if {c.name} else \"0\"" if lite else f"\"true\" if {c.name} else \"false\"" return f"str({c.name})" fn camel(name: str) str: out = "" up = true for ch in name: if ch == "_": up = true elif up: out += ch.upper() up = false else: out += ch return out # a Tern string literal fn quote(s: str) str: return "\"" + s.replace("\\", "\\\\").replace("\"", "\\\"").replace("\t", "\\t").replace("\n", "\\n") + "\"" # A query that returns a whole table's columns, in order (SELECT * FROM users), gets the # table's struct (User), shared by every such query, instead of one of its own. fn use_models(dbase: Database, ds: mut [Described]): table_cols: {str: [str]} = {} names: {str: str} = {} # struct name -> table for i in 0..len(ds): d = ds[i] if d.q.cmd == "exec" or len(d.cols) < 2: continue t = d.cols[0].table if t == "" or len(d.cols.filter(fn(c) c.table != t)) > 0: continue if not (t in table_cols): table_cols[t] = table_columns(dbase, t) if d.cols.map(fn(c) c.name) != table_cols[t]: continue name = model_name(t) if name in names and names[name] != t: name = camel(t) + "Table" names[name] = t ds[i].model = name fn table_columns(dbase: Database, table: str) [str]: match dbase: Postgres(p): rows = p.query("SELECT attname::text AS n FROM pg_attribute WHERE attrelid = $1::regclass AND attnum > 0 AND NOT attisdropped ORDER BY attnum", [table]) return rows.map(fn(r) r["n"]) Sqlite(d): rows = d.query("SELECT name AS n FROM pragma_table_info(?) ORDER BY cid", [table]) return rows.map(fn(r) r["n"]) # users -> User, people -> Person, categories -> Category, addresses -> Address, user_roles -> UserRole fn model_name(table: str) str: t = table if t == "people": t = "person" elif t.ends_with("ies") and len(t) > 3: t = t[..len(t) - 3] + "y" elif t.ends_with("sses") or t.ends_with("xes") or t.ends_with("ches") or t.ends_with("shes"): t = t[..len(t) - 2] elif t.ends_with("s") and not t.ends_with("ss") and not t.ends_with("us"): t = t[..len(t) - 1] return camel(t) fn helper_suffix(c: Col) str: # Option[[int]] -> list_int; a timestamp: float_time ty = c.ty[7..len(c.ty) - 1] if c.ty.starts_with("Option[") else c.ty return ty.replace("[", "list_").replace("]", "") + ("_time" if is_time(c.db_type) else "") fn is_time(pg_type: str) bool: return pg_type == "timestamptz" or pg_type == "timestamp" ``` ## deps/sqlgen/enums.tn ```tern # sqlgen: PostgreSQL enum types as Tern enums (see sqlgen.tn) # # CREATE TYPE order_status AS ENUM ('pending', 'in_progress', 'shipped'); # # becomes # # enum OrderStatus: # Pending # InProgress # Shipped # # fn order_status_label(v: OrderStatus) str "in_progress" for InProgress # # Rows read the labels into the enum (pg.fetch matches a label to its variant); parameters # of the type are sent as their label. struct EnumType: oid: int pg_name: str # order_status name: str # OrderStatus labels: [str] # as PostgreSQL has them, in order variants: [str] # the variant for each label fn enum_name(pg_type: str) str: return camel(ident_of(pg_type)) fn label_fn(pg_type: str) str: return ident_of(pg_type) + "_label" # anything but letters and digits -> _, lower case fn ident_of(s: str) str: out = "" for ch in s.lower(): c = ch.byte(0) out += ch if ident_byte(c) else "_" if out == "" or is_digit(out.byte(0)): out = "t_" + out return out # "in_progress", "in progress", "In-Progress" -> InProgress (and it must read back: the # decoder matches labels to variants ignoring case, _, - and spaces) fn variant_name(label: str) Result[str, str]: out = "" up = true for ch in label: c = ch.byte(0) if ident_byte(c) and c != 95: out += ch.upper() if up else ch up = false else: up = true loose = label.lower().replace("_", "").replace("-", "").replace(" ", "") if out == "" or loose != out.lower(): return Err(f"the label '{label}' can't be a variant name (use letters, digits, _, - and spaces)") if is_digit(out.byte(0)): return Err(f"the label '{label}' can't be a variant name (it starts with a digit)") return Ok(out) # the enum types the queries use, with their labels fn find_enums(dbase: Database, ds: [Described]) Result[[EnumType], str]: match dbase: Postgres(p): return _pg_enums(p, ds) Sqlite(d): none: [EnumType] = [] return Ok(none) fn _pg_enums(p: pg.Pool, ds: [Described]) Result[[EnumType], str]: out: [EnumType] = [] seen: {int: bool} = {} for d in ds: cols = d.cols for c in d.params: cols.push(c) for c in cols: if c.enum_oid == 0 or c.enum_oid in seen: continue seen[c.enum_oid] = true rows = p.query("SELECT enumlabel::text AS l FROM pg_enum WHERE enumtypid = $1::oid ORDER BY enumsortorder", [str(c.enum_oid)]) labels = rows.map(fn(r) r["l"]) e = EnumType(c.enum_oid, c.db_type, enum_name(c.db_type), labels, []) taken: {str: str} = {} for l in labels: v = variant_name(l).map_err(fn(x) f"sqlgen: enum {c.db_type}: {x}")? if v in taken: return Err(f"sqlgen: enum {c.db_type}: the labels '{taken[v]}' and '{l}' both become {v}") taken[v] = l e.variants.push(v) out.push(e) return Ok(out) fn emit_enum(e: EnumType) str: out = f"\n# the PostgreSQL enum {e.pg_name}\nenum {e.name}:\n" for v in e.variants: out += f" {v}\n" out += f"\n# the label PostgreSQL knows a {e.name} by\nfn {label_fn(e.pg_name)}(v: {e.name}) str:\n match v:\n" for i, v in e.variants: out += f" {v}:\n return {quote(e.labels[i])}\n" return out ``` ## deps/sqlgen/infer.tn ```tern # sqlgen: SQLite doesn't know its parameters' types, so they're read from where each one is # used: `col = ?1` (and <, >, <=, >=, <>, !=, LIKE) takes col's type, INSERT INTO t (a, b) # VALUES (?1, ?2) takes a's and b's, LIMIT / OFFSET take int. Anything else is str unless # written @name:type. # the table columns' declared types: "users.id" and (when no two tables disagree) "id" fn column_types(d: sqlite.Db) {str: str}: out: {str: str} = {} clash: {str: bool} = {} rows = d.query("SELECT m.name AS tbl, p.name AS col, p.type AS type FROM sqlite_master m, pragma_table_info(m.name) p WHERE m.type IN ('table', 'view')", []) for r in rows: ty = sqlite_type(r["type"]) out[r["tbl"] + "." + r["col"]] = ty c = r["col"] if c in out and out[c] != ty: clash[c] = true out[c] = ty for c in clash.keys(): out.remove(c) return out # a declared type -> a Tern type, by SQLite's affinity rules (plus BOOL) fn sqlite_type(decl: str) str: d = decl.upper() if d.contains("INT"): return "int" if d.contains("BOOL"): return "bool" if d.contains("REAL") or d.contains("FLOA") or d.contains("DOUB") or d.contains("NUMERIC") or d.contains("DECIMAL"): return "float" return "str" # the types of ?1 .. ?n ("" where nothing says) fn infer_params(sql: str, n: int, cols: {str: str}) [str]: out = [""] * n ts = _tokens(sql) for i, t in ts: p = _param_index(t) if p < 0 or p >= n or out[p] != "": continue prev = ts[i - 1].upper() if i > 0 else "" if prev == "LIMIT" or prev == "OFFSET": out[p] = "int" elif i >= 2 and _is_compare(prev): out[p] = _type_of(cols, ts[i - 2]) elif i + 2 < len(ts) and _is_compare(ts[i + 1].upper()): out[p] = _type_of(cols, ts[i + 2]) _infer_insert(ts, cols, &out) return out # INSERT INTO t (a, b, ...) VALUES (?1, ?2, ...), row by row fn _infer_insert(ts: [str], cols: {str: str}, out: mut [str]): i = 0 while i + 2 < len(ts): if ts[i].upper() == "INSERT": k = i + 1 while k < len(ts) and ts[k].upper() != "INTO": k += 1 table = ts[k + 1] if k + 1 < len(ts) else "" k += 2 names: [str] = [] if k < len(ts) and ts[k] == "(": k += 1 while k < len(ts) and ts[k] != ")": if ts[k] != ",": names.push(copy(ts[k])) k += 1 k += 1 if k < len(ts) and ts[k].upper() == "VALUES": k += 1 while k < len(ts) and ts[k] == "(": pos = 0 k += 1 depth = 0 while k < len(ts) and not (ts[k] == ")" and depth == 0): if ts[k] == "(": depth += 1 elif ts[k] == ")": depth -= 1 elif ts[k] == "," and depth == 0: pos += 1 elif depth == 0 and pos < len(names): p = _param_index(ts[k]) if p >= 0 and p < len(out) and out[p] == "": out[p] = _type_of(cols, table + "." + names[pos]) k += 1 k += 1 if k < len(ts) and ts[k] == ",": k += 1 i = k else: i += 1 fn _type_of(cols: {str: str}, ident: str) str: if ident in cols: return cols[ident] dot = ident.find(".") if dot >= 0 and ident[dot + 1..] in cols: return cols[ident[dot + 1..]] return "" fn _is_compare(t: str) bool: return t in ["=", "==", "<>", "!=", "<", ">", "<=", ">=", "LIKE", "GLOB", "IS"] # "?3" -> 2; anything else -> -1 fn _param_index(t: str) int: if len(t) < 2 or t[0] != "?": return -1 return t[1..].to_int(0) - 1 # identifiers (a.b kept together), ?N parameters, operators and punctuation; strings, # numbers and comments become "0" fn _tokens(sql: str) [str]: out: [str] = [] top = 0 i = 0 n = len(sql) while i < n: c = sql.byte(i) j = i + 1 if c == 32 or c == 9 or c == 10 or c == 13: i = j continue if c == 39: while j < n and sql.byte(j) != 39: j += 1 out.push("0") i = j + 1 continue if c == 45 and j < n and sql.byte(j) == 45: while j < n and sql.byte(j) != 10: j += 1 i = j continue if ident_start(c) or c == 34 or c == 96 or c == 91: # names: a, "a", `a`, [a], a.b name = "" k = i while k < n: ch = sql.byte(k) if ch == 34 or ch == 96 or ch == 91: close = 93 if ch == 91 else ch e = k + 1 while e < n and sql.byte(e) != close: e += 1 name += sql[k + 1..e] k = e + 1 elif ident_byte(ch): s = k while k < n and ident_byte(sql.byte(k)): k += 1 name += sql[s..k] else: break if k < n and sql.byte(k) == 46: name += "." k += 1 else: break out.push(name) i = k continue if c == 63: # ?N, or ?: one more than the highest so far while j < n and is_digit(sql.byte(j)): j += 1 num = sql[i + 1..j].to_int(0) if j > i + 1 else top + 1 top = max(top, num) out.push("?" + str(num)) i = j continue if is_digit(c): while j < n and (is_digit(sql.byte(j)) or sql.byte(j) == 46): j += 1 out.push("0") i = j continue two = sql[i..i + 2] if two in ["<=", ">=", "<>", "!=", "=="]: out.push(two) i += 2 continue out.push(sql[i]) i = j return out ``` ## deps/sqlgen/queries.tn ```tern # sqlgen: reading a queries file (see sqlgen.tn) struct Query: name: str # the function to generate cmd: str # "one", "many", or "exec" raw: str # the SQL as written docs: [str] # the comment lines under `-- name:` file: str line: int # a query made ready for one database struct Prepared: sql: str # @name parameters turned into $1, $2 (PostgreSQL) or ?1, ?2 (SQLite) params: [str] # the @names, in order ([] when the SQL numbers its own) param_types: {str: str} # from @name:type nullable: {str: bool} # from @name? (an Option, NULL when None) forced: {str: bool} # column -> nullable, from "col!" (false) and "col?" (true) col_types: {str: str} # column -> type, from "col: type" # the queries in a file: each starts with `-- name: NAME :one|:many|:exec` fn parse_queries(file: str, text: str) Result[[Query], str]: out: [Query] = [] cur: [str] = [] head = "" start = 0 lines = text.replace("\r\n", "\n").split("\n") for i, line in lines: t = line.trim() if t.starts_with("-- name:"): if head != "": out.push(_query(file, start, head, cur)?) elif _has_sql(cur): return Err(f"{file}:{start + 1}: SQL before the first `-- name: NAME :one|:many|:exec` line") head = t start = i + 1 cur = [] else: cur.push(line) if head != "": out.push(_query(file, start, head, cur)?) seen: {str: int} = {} for q in out: if q.name in seen: return Err(f"{file}:{q.line}: a query named {q.name} is already on line {seen[q.name]}") seen[q.name] = q.line return Ok(out) fn _query(file: str, line: int, head: str, body: [str]) Result[Query, str]: where = f"{file}:{line}" words = head[8..].trim().split(" ").filter(fn(w) w != "") if len(words) != 2 or not words[1].starts_with(":"): return Err(f"{where}: write `-- name: NAME :one` (or :many, :exec), not `{head}`") name = words[0] cmd = words[1][1..] if not (cmd in ["one", "many", "exec"]): return Err(f"{where}: {name}: `:{cmd}` should be :one (at most one row), :many (every row), or :exec (no rows)") if not is_ident(name) or name in reserved() or name in keywords(): return Err(f"{where}: `{name}` can't name a function; use a name like get_user") docs: [str] = [] k = 0 while k < len(body) and body[k].trim().starts_with("--"): docs.push(body[k].trim()[2..].trim()) k += 1 sql = _code(body[k..]).trim() while sql.ends_with(";"): sql = sql[..len(sql) - 1].trim() if sql == "": return Err(f"{where}: {name} has no SQL") if len(migrate.split_statements(sql)) > 1: return Err(f"{where}: {name} has more than one statement; give each its own `-- name:`") return Ok(Query(name, cmd, sql, docs, file, line)) fn _code(lines: [str]) str: return "\n".join(lines) fn _has_sql(lines: [str]) bool: # anything but blank lines and -- comments for l in lines: t = l.trim() if t != "" and not t.starts_with("--"): return true return false fn tern_types() [str]: return ["int", "float", "bool", "str"] # @name[?][:type] parameters -> $N (mark "$") or ?N (mark "?"), the same name the same number; # "col!" / "col?" / "col: type" column names -> "col", remembering what they said. Quotes, # $$ bodies and comments are left alone. fn prepare(q: Query, mark: str) Result[Prepared, str]: where = f"{q.file}:{q.line}: {q.name}" sql = q.raw pr = Prepared("", [], {}, {}, {}, {}) out = "" i = 0 n = len(sql) numbered = false while i < n: c = sql.byte(i) j = i + 1 if c == 39: # '...' while j < n and sql.byte(j) != 39: j += 1 j += 1 elif c == 34: # "identifier", maybe with a note while j < n and sql.byte(j) != 34: j += 1 col = _column_note(&pr, sql[i + 1..j], where)? out += "\"" + col + "\"" i = j + 1 continue elif c == 45 and j < n and sql.byte(j) == 45: # -- comment while j < n and sql.byte(j) != 10: j += 1 elif c == 47 and j < n and sql.byte(j) == 42: # /* comment */ end = sql[j..].find("*/") j = n if end < 0 else j + end + 2 elif c == 36 and j < n and is_digit(sql.byte(j)): # $1 numbered = true while j < n and is_digit(sql.byte(j)): j += 1 elif c == 63 and mark == "?": # ? or ?1 (SQLite; jsonb operators in PostgreSQL) numbered = true elif c == 36 and mark == "$": # $tag$ ... $tag$ k = j while k < n and ident_byte(sql.byte(k)): k += 1 if k < n and sql.byte(k) == 36: tag = sql[i..k + 1] end = sql[k + 1..].find(tag) j = n if end < 0 else k + 1 + end + len(tag) elif c == 64 and j < n and ident_start(sql.byte(j)) and not _operator_before(sql, i): # @name while j < n and ident_byte(sql.byte(j)): j += 1 pname = sql[i + 1..j] if pname in keywords() or pname in ["pool", "conn"]: return Err(f"{where}: rename the parameter @{pname}") if j < n and sql[j] == "?": # @name? pr.nullable[pname] = true j += 1 if j + 1 < n and sql[j] == ":" and sql[j + 1] != ":": # @name:type k = j + 1 while k < n and ident_byte(sql.byte(k)): k += 1 ty = sql[j + 1..k] if not (ty in tern_types()): return Err(f"{where}: @{pname}:{ty}: the type is int, float, bool, or str") pr.param_types[pname] = ty j = k k = index_of(pr.params, pname) if k < 0: pr.params.push(copy(pname)) k = len(pr.params) - 1 out += mark + str(k + 1) i = j continue out += sql[i..j] i = j if numbered and len(pr.params) > 0: return Err(f"{where}: use @name parameters or numbered ones, not both") pr.sql = out return Ok(pr) # "total!", "email?", "n: int", "n!: int" -> the column name, remembering the rest fn _column_note(pr: mut Prepared, inner: str, where: str) Result[str, str]: col = inner ty = "" colon = inner.find(":") if colon > 0: ty = inner[colon + 1..].trim() if not (ty in tern_types()): return Err(f"{where}: \"{inner}\": the type after : is int, float, bool, or str") col = inner[..colon].trim() if len(col) > 1 and (col.ends_with("!") or col.ends_with("?")): pr.forced[col[..len(col) - 1]] = col.ends_with("?") col = col[..len(col) - 1] if ty != "": pr.col_types[col] = ty return Ok(col) # `a <@ b`, `@@`, `x@y`: operators and other text, not parameters fn _operator_before(sql: str, i: int) bool: return i > 0 and (ident_byte(sql.byte(i - 1)) or "<>@#~!=-+*/%^&|`?".contains(sql[i - 1])) fn index_of(xs: [str], x: str) int: for i, y in xs: if y == x: return i return -1 fn is_digit(c: int) bool: return c >= 48 and c <= 57 fn ident_start(c: int) bool: return (c >= 65 and c <= 90) or (c >= 97 and c <= 122) or c == 95 fn ident_byte(c: int) bool: return ident_start(c) or is_digit(c) fn is_ident(s: str) bool: if s == "" or not ident_start(s.byte(0)): return false for i in 1..len(s): if not ident_byte(s.byte(i)): return false return true ``` ## deps/sqlgen/sqlgen.tn ```tern # sqlgen: typed Tern functions from SQL queries, checked against a PostgreSQL or SQLite # database (like Go's sqlc). # # You write the queries, in SQL, each under a `-- name:` line (queries.sql): # # -- name: get_user :one # SELECT id, name, email FROM users WHERE id = @id; # # -- name: list_users :many # -- Every user, newest first. # SELECT id, name FROM users ORDER BY id DESC LIMIT @max; # # -- name: rename_user :exec # UPDATE users SET name = @name WHERE id = @id; # # sqlgen asks the database for each query's parameter and column types, and writes a # module (users_db.tn) with a struct per result and a function per query: # # struct GetUserRow: # id: int # name: str # email: Option[str] # the column can be NULL # # PostgreSQL (import pg): # fn get_user(pool: pg.Pool, id: int) Result[Option[GetUserRow], str] # fn list_users(pool: pg.Pool, max: int) Result[[ListUsersRow], str] # fn rename_user(pool: pg.Pool, name: str, id: int) Result[int, str] (rows changed) # SQLite (import sqlite; its errors are runtime errors, so no Result): # fn get_user(conn: sqlite.Db, id: int) Option[GetUserRow] # fn list_users(conn: sqlite.Db, max: int) [ListUsersRow] # fn rename_user(conn: sqlite.Db, name: str, id: int) int (rows changed) # # so a typo in a column, a missing parameter, or a nullable column read as if it weren't is # an error when you generate (or `tern check`), not when the query runs. # # Running it: a program of your own (gen.tn), # # import os # import sqlgen # # fn main() Result[int, str]: # return sqlgen.cli(os.env("DATABASE_URL", "postgres://postgres@localhost/app")) # or sqlite:app.db # # tern run gen.tn -- queries.sql -o users_db.tn against the database as it is # tern run gen.tn -- queries.sql -o users_db.tn --migrations migrations # in a scratch database made from your migrations (see migrate) and # dropped afterwards: no schema needed anywhere, good for CI # tern run gen.tn -- ... --check fail if users_db.tn isn't what sqlgen would write # # Functions: # sqlgen.cli(url str) Result[int, str] the command above (arguments from os.args()) # sqlgen.command(url str, args [str]) Result[int, str] ... with the arguments given # sqlgen.postgres(p pg.Pool) Database sqlgen.sqlite(d sqlite.Db) Database # sqlgen.generate(dbase Database, files [str]) Result[str, str] the module's text # sqlgen.parse_queries(file str, text str) Result[[Query], str] # # Queries: # -- name: NAME :one the first row, or None # -- name: NAME :many every row # -- name: NAME :exec the number of rows changed # Comment lines right under the name line become the function's comment. Parameters are # @name (the same name twice is one parameter), or numbered ($1 in PostgreSQL, ?1 or ? # in SQLite: named arg1, arg2, ...). A query returning one column gives that column's # values, not a Row. # A query returning every column of one table, in order (SELECT * FROM users), uses the # table's struct (User), shared by all such queries. # # Types: PostgreSQL's int2/int4/int8/oid -> int, float4/float8/numeric -> float, bool -> bool, # timestamptz/timestamp -> float (seconds since 1970, UTC, like time.now()), arrays of those # (and of text, varchar, uuid) -> [int], [float], [bool], [str], the rest (text, date, uuid, # json, enums) -> str, as PostgreSQL writes them. # SQLite goes by the declared type (INT... -> int, REAL/FLOAT/DOUBLE/NUMERIC -> float, # BOOL... -> bool, else str); an expression is str unless its name says `"n: int"`. SQLite # parameters take the type of the column they're compared with or inserted into (LIMIT and # OFFSET: int), else str; `@id:int` says it outright. # Enums: a PostgreSQL enum type becomes a Tern enum (order_status -> OrderStatus, labels # in_progress -> InProgress) and a function order_status_label(v) str; columns and parameters # of the type use it. # Nulls: a column is an Option unless it is a table column declared NOT NULL (with a LEFT / # RIGHT / FULL join, every column is). Say otherwise in its name: `count(*) AS "total!"` # (never null), `u.name AS "name?"` (maybe null), `count(*) AS "n!: int"` (SQLite). # A parameter that may be NULL is written @x? (Option[T] in the function; None is NULL). import crypto import sqlite import fs import migrate import os import pg fn postgres(p: pg.Pool) Database: return Database.Postgres(p) fn sqlite(d: sqlite.Db) Database: return Database.Sqlite(d) fn generate(dbase: Database, files: [str]) Result[str, str]: cols_of: {str: str} = {} if dbase is Sqlite(d): cols_of = column_types(d) ds: [Described] = [] names: {str: str} = {} for f in files: text = fs.try_read(f).context("sqlgen")? qs = parse_queries(f, text)? for q in qs: if q.name in names: return Err(f"{q.file}:{q.line}: {q.name} is also in {names[q.name]}") names[q.name] = q.file ds.push(describe_query(dbase, q, cols_of)?) use_models(dbase, &ds) enums = find_enums(dbase, ds)? return Ok(emit(dbase, ds, enums, ", ".join(files.map(fn(f) _base(f))))) fn _base(path: str) str: parts = path.split("/") return parts[len(parts) - 1] # QUERIES.sql ... -o OUT.tn [--migrations DIR] [--check] fn cli(url: str) Result[int, str]: return command(url, os.args()[1..]) # the same, with the arguments given (`tern sqlgen` calls it) fn command(url: str, args: [str]) Result[int, str]: usage = "usage: QUERIES.sql ... -o OUT.tn [--migrations DIR] [--check]" files: [str] = [] out = "" migrations = "" check = false i = 0 while i < len(args): a = args[i] if a == "-o" and i + 1 < len(args): out = args[i + 1] i += 1 elif a == "--migrations" and i + 1 < len(args): migrations = args[i + 1] i += 1 elif a == "--check": check = true elif a.ends_with(".sql"): files.push(a) else: return Err(f"sqlgen: unexpected `{a}`; {usage}") i += 1 if len(files) == 0 or out == "": return Err("sqlgen: " + usage) code = generate_at(url, files, migrations)? if check: if fs.read(out) != code: return Err(f"sqlgen: {out} is out of date: run sqlgen again") print(f"{out} is up to date") return Ok(0) fs.try_write(out, code).context("sqlgen")? print(f"wrote {out}") return Ok(0) # generate() against the database at url (postgres://..., sqlite:PATH), or with migrations # given, against a scratch database made from them fn generate_at(url: str, files: [str], migrations: str) Result[str, str]: if not (url.starts_with("postgres://") or url.starts_with("postgresql://")): path = url[7..] if url.starts_with("sqlite:") else url if migrations == "": if not fs.exists(path): return Err(f"sqlgen: no database at {path}") return generate(Database.Sqlite(sqlite.open(path)), files) dir = fs.temp_dir("sqlgen") d = sqlite.open(dir + "/scratch.db") r = _generate_migrated(Database.Sqlite(d), migrate.sqlite(d), files, migrations) fs.remove_all(dir) return r p = pg.pool(url, 2) if not p.ok(): return Err("sqlgen: can't connect: " + pg.error()) if migrations == "": r = generate(Database.Postgres(p), files) p.close() return r name = "sqlgen_" + crypto.hex(crypto.random_bytes(6)) made = p.run_script(f"CREATE DATABASE {name}").context("sqlgen: making a scratch database") if made is Err(e): p.close() return Err(e) scratch = pg.pool(with_database(url, name), 2) r: Result[str, str] = Err("sqlgen: can't connect to the scratch database: " + pg.error()) if scratch.ok(): r = _generate_migrated(Database.Postgres(scratch), migrate.postgres(scratch), files, migrations) scratch.close() p.run_script(f"DROP DATABASE {name} WITH (FORCE)") p.close() return r fn _generate_migrated(dbase: Database, t: migrate.Target, files: [str], migrations: str) Result[str, str]: t.up(migrations)? return generate(dbase, files) # the same server, another database: postgres://user:pass@host:port/NAME?options fn with_database(url: str, name: str) str: at = url.find("//") + 2 slash = url[at..].find("/") q = url[at..].find("?") if slash < 0: if q < 0: return url + "/" + name return url[..at + q] + "/" + name + url[at + q..] rest = url[at + slash + 1..] k = rest.find("?") return url[..at + slash + 1] + name + ("" if k < 0 else rest[k..]) ``` # Example programs ## examples/01_values.tn ```tern # Values: assignment copies. No pointers, no lifetimes, no garbage collector. struct Point: x, y: int struct Shape: name: str points: [Point] fn scale(p: mut Point, k: int): # `mut`: the only way to change the caller's value p.x *= k p.y *= k fn size(p: Point) int: # parameters are read-only by default return abs(p.x) + abs(p.y) fn count(s: take Shape) int: # `take`: ownership moves in, nothing is copied return len(s.points) fn main(): a = Point(3, -4) b = a # a copy scale(&b, 10) # `&` shows the change at the call site print("a =", a, "b =", b, "size(b) =", size(b)) big = [1, 2, 3, 4, 5, 6, 7, 8] snapshot = big # instant: storage is shared until someone writes big.push(9) # first write while shared makes one copy big.push(10) # already unique: no copy print("snapshot:", snapshot) print("big: ", big) s = Shape(name="tri", points=[a, b, Point(x=0, y=0)]) s.points[2].x = 7 # nested update, still no pointers t = s s.points[0].y = 99 # t is unaffected print(t) print(s) total = 0 for p in s.points: total += size(p) print("total size:", total, "points:", count(s)) ``` ## examples/02_parallel.tn ```tern # `parallel:` runs each line as its own task and waits for all of them. # Tasks can read anything; a variable written by one task can't be touched by another. fn sum(xs: [int], lo: int, hi: int) int: total = 0 for i in lo..hi: total += xs[i] return total fn main(): data: [int] = [] for i in 0..1000: data.push(i * i) half = len(data) / 2 parallel: left = sum(data, 0, half) # reads data, writes left right = sum(data, half, len(data)) # reads data, writes right doubled = [0] * len(data) parallel for i in 0..len(data): # every iteration can run on its own core doubled[i] = data[i] * 2 # ok: iteration i only writes doubled[i] print("left + right =", left + right, " check =", sum(data, 0, len(data))) print("doubled[999] =", doubled[999]) ``` ## examples/06_unique.tn ```tern # The optimizer must never change what a program means: shared arrays still copy on write. fn leak(src: mut [int], dst: mut [int]): dst = src # src escapes into dst fn fill(xs: mut [int]): for i in 0..len(xs): xs[i] = i * 10 fn same(xs: [int]) [int]: return xs # a read-only parameter escaping through the return value fn main(): a = [0] * 3 b = [0] * 3 leak(&a, &b) # a and b now share storage a[0] = 42 # must copy: b stays [0, 0, 0] print(a, b) c = [1, 2, 3] # provably never shared: no checks at all fill(&c) print(c) d = [7, 8, 9] e = same(d) # d escapes through the return value d[0] = 0 print(d, e) evens: [int] = [] for i in 0..20: if i % 2 == 1: continue if i > 12: break evens.push(i) print(evens, min(3, 9), max(2.5, 1.0)) ``` ## examples/web/hello.tn ```tern # The smallest Tern web server. import http fn handle(req: http.Request) http.Response: return http.text("Hello, World!") fn main(): http.serve(8080, handle) ``` ## examples/web/api.tn ```tern # A small JSON API. The app data is shared read-only by every server thread, # so requests run fully in parallel with no locks. import http struct User: id: int name: str email: str tags: [str] struct App: users: {int: User} greeting: str fn user_json(u: User) str: tags = [] for t in u.tags: tags.push(json_quote(t)) return f"{{\"id\": {u.id}, \"name\": {json_quote(u.name)}, \"email\": {json_quote(u.email)}, \"tags\": [{", ".join(tags)}]}}" fn handle(req: http.Request, app: App) http.Response: if req.path == "/": return http.html("

Tern API

Try /users or /users/2

") if req.path == "/hello": name = req.query("name") if name == "": name = "world" return http.text(f"{app.greeting}, {name}!") if req.path == "/users": items = [] for id, u in app.users: items.push(user_json(u)) return http.json(f"[{", ".join(items)}]") if req.path.starts_with("/users/"): id = req.path[len("/users/")..].to_int(-1) if id in app.users: return http.json(user_json(app.users[id])) return http.status(404, f"no user {id}") return http.not_found() fn main(): users: {int: User} = {} users[1] = User(1, "Ada Lovelace", "ada@example.com", ["admin", "math"]) users[2] = User(2, "Grace Hopper", "grace@example.com", ["compilers"]) users[3] = User(3, "Linus", "linus@example.com", []) http.serve(8080, handle, App(users, "Hello")) ``` ## examples/web/todo.tn ```tern # A todo API with shared, changing state. The handler takes `app: mut App`, so the # server runs these requests one at a time: no data races, no locks to write. import http struct Todo: id: int title: str done: bool struct App: todos: {int: Todo} next_id: int fn todo_json(t: Todo) str: return f"{{\"id\": {t.id}, \"title\": {json_quote(t.title)}, \"done\": {t.done}}}" fn handle(req: http.Request, app: mut App) http.Response: if req.path == "/todos" and req.method == "GET": items = [] for id, t in app.todos: items.push(todo_json(t)) return http.json(f"[{", ".join(items)}]") if req.path == "/todos" and req.method == "POST": title = req.body.trim() if title == "": return http.status(400, "empty title") id = app.next_id app.next_id += 1 app.todos[id] = Todo(id, title, false) return http.Response(201, "application/json", todo_json(app.todos[id]), http.header_line("Location", f"/todos/{id}"), 0) if req.path.starts_with("/todos/"): id = req.path[len("/todos/")..].to_int(-1) if not (id in app.todos): return http.status(404, f"no todo {id}") if req.method == "PUT": app.todos[id].done = true return http.json(todo_json(app.todos[id])) if req.method == "DELETE": app.todos.remove(id) return http.status(204, "") return http.json(todo_json(app.todos[id])) return http.not_found() fn main(): app = App({}, 1) http.serve(8081, handle, &app) ``` ## tests/run/enums.tn ```tern import math enum Shape: Circle(radius: float) Rect(w: float, h: float) Empty enum Expr: # recursive: Add/Mul hold Exprs (boxed automatically) Num(v: int) Var(name: str) Add(l: Expr, r: Expr) Mul(l: Expr, r: Expr) enum Color: Red Green Blue struct Scene: name: str shapes: [Shape] fn area(s: Shape) float: match s: Circle(r): return math.pi() * r * r Rect(w, h): return w * h Empty: return 0.0 fn eval(e: Expr, env: {str: int}) int: match e: Num(v): return v Var(name): return env.get(name, 0) Add(l, r): return eval(l, env) + eval(r, env) Mul(l, r): return eval(l, env) * eval(r, env) fn show(e: Expr) str: match e: Num(v): return str(v) Var(name): return name Add(l, r): return f"({show(l)} + {show(r)})" Mul(l, r): return f"{show(l)} * {show(r)}" fn status_text(code: int) str: match code: 200: return "ok" 404: return "not found" _: return "other" fn method_kind(m: str) str: match m: "GET": return "read" "POST": return "write" _: return "?" fn main(): shapes = [Shape.Circle(1.0), Shape.Rect(w=2.0, h=3.0), Shape.Empty] total = 0.0 for s in shapes: total += area(s) print(math.round(total * 1000.0) / 1000.0, shapes) scene = Scene("demo", shapes) copy_ = scene scene.shapes[0] = Shape.Rect(1.0, 1.0) print(copy_.shapes[0], scene.shapes[0]) e = Expr.Add(Expr.Num(2), Expr.Mul(Expr.Var("x"), Expr.Num(10))) print(show(e), "=", eval(e, {"x": 4})) deep = Expr.Num(0) for i in 1..2000: deep = Expr.Add(deep, Expr.Num(i)) print(eval(deep, {})) c = Color.Green print(c == Color.Green, c != Color.Blue, c) print(status_text(404), status_text(500), method_kind("POST"), method_kind("PUT")) n = 0 for s in shapes: match s: Empty: break _: n += 1 print(n) ``` Output: ```output 9.142 [Shape.Circle(radius: 1.0), Shape.Rect(w: 2.0, h: 3.0), Shape.Empty] Shape.Circle(radius: 1.0) Shape.Rect(w: 1.0, h: 1.0) (2 + x * 10) = 42 1999000 true true Color.Green not found other write ? 2 ``` ## tests/run/closures.tn ```tern # Function values and closures: captures are copies made when the closure is created, # so closures are plain values (shareable across threads, no cycles, no leaks). struct Point: x, y: int struct Button: label: str on_click: fn(int) str fn double(x: int) int: return x * 2 fn apply(f: fn(int) int, x: int) int: return f(x) fn twice(f: fn(int) int) fn(int) int: return fn(x) f(f(x)) # a closure capturing a closure fn make_counter_label(prefix: str) fn(int) str: fn label(n: int) str: if n == 1: return f"{prefix}: one" return f"{prefix}: {n}" return label fn scale(p: mut Point, k: int): p.x *= k p.y *= k fn main(): # named functions as values print(apply(double, 21), apply(twice(double), 3)) # capture happens at creation k = 10 add = fn(x: int) x + k k = 99 print(add(5), k) # heap captures: strings and arrays (a later change doesn't reach the copy) xs = [1, 2, 3] total = fn() len(xs) xs.push(4) xs[0] = 100 print(total(), xs) # closures in structs, arrays, and maps who = "tern" b = Button("go", fn(n) f"{who} clicked {n} times") print(b.label, b.on_click(3)) ops: {str: fn(int, int) int} = {"add": fn(a, b) a + b, "mul": fn(a, b) a * b} for name, op in ops: print(name, op(6, 7)) fs: [fn(int) int] = [double, add, fn(x) x * x] out: [int] = [] for f in fs: out.push(f(7)) print(out, fs[2](5)) # nested named functions (recursion, captures) base = 3 fn power(n: int) int: if n == 0: return 1 return base * power(n - 1) print(power(4), make_counter_label("items")(1), make_counter_label("items")(5)) # `mut` and `take` parameters grow: fn(mut Point) = fn(p) scale(&p, 3) p = Point(1, 2) grow(&p) print(p) keep: fn(take [str]) int = fn(words) len(words) ws = ["a", "b"] print(keep(ws)) # copying and reassigning (the old closure is released) g = copy(add) add = fn(x: int) x - 1 print(g(1), add(1)) # closures are values: shared by parallel tasks sq = fn(x: int) x * x r1 = 0 r2 = 0 parallel: r1 = sq(12) r2 = sq(13) ys = [0] * 8 parallel for i in 0..8: ys[i] = sq(i) + k print(r1, r2, ys) print(sq) ``` Output: ```output 42 12 15 99 3 [100, 2, 3, 4] go tern clicked 3 times add 13 mul 42 [14, 17, 49] 25 81 items: one items: 5 Point{x: 3, y: 6} 2 11 0 144 169 [99, 100, 103, 108, 115, 124, 135, 148] ``` ## tests/run/strings.tn ```tern struct User: name: str tags: [str] fn greet(u: User) str: return f"Hello, {u.name}! You have {len(u.tags)} tags: {", ".join(u.tags)}" fn main(): s = "hello" t = s + " world" print(t, len(t), t[0], t[6..], t[..5], t[1..4]) print(t.find("world"), "wor" in t, t.starts_with("he"), t.ends_with("x")) print(" padded\t".trim(), "MiXeD".upper(), "MiXeD".lower(), "a-b-c".replace("-", "+")) parts = "GET /users/42 HTTP/1.1".split(" ") print(parts, len(parts), parts[1].split("/")) id = parts[1].split("/")[2].to_int(-1) print(id + 1, "abc".to_int(-1), "3.5".to_float(0.0) * 2.0) u = User("ada", ["admin", "dev"]) print(greet(u)) v = u v.name = "bob" print(u.name, v.name, u) out = "" for i in 0..5: out += str(i) + "," print(out, "x" * 0 == "" ) ``` Output: ```output hello world 11 h world hello ell 6 true true false padded MIXED mixed a+b+c ["GET", "/users/42", "HTTP/1.1"] 3 ["", "users", "42"] 43 -1 7.0 Hello, ada! You have 2 tags: admin, dev ada bob User{name: "ada", tags: ["admin", "dev"]} 0,1,2,3,4, true ``` ## tests/run/maps.tn ```tern struct Store: users: {int: str} counts: {str: int} fn count_words(text: str) {str: int}: counts: {str: int} = {} for w in text.split(" "): if w == "": continue counts[w] = counts.get(w, 0) + 1 return counts fn add(s: mut Store, id: int, name: str): s.users[id] = name s.counts[name] = s.counts.get(name, 0) + 1 fn main(): ages = {"ada": 36, "bob": 25} ages["cy"] = 41 ages["bob"] += 1 print(ages, len(ages), "ada" in ages, "zed" in ages, ages.get("zed", -1)) for name, age in ages: print(name, age) snapshot = ages ages.remove("ada") print(ages, snapshot, ages.keys(), snapshot.values()) wc = count_words("the cat and the hat and the bat") print(wc) for i, w in ["a", "b", "c"]: print(i, w) st = Store({}, {}) add(&st, 1, "ada") add(&st, 2, "bob") add(&st, 3, "ada") print(st) big: {int: int} = {} for i in 0..100_000: big[i] = i * 2 for i in 0..100_000: if i % 3 == 0: big.remove(i) total = 0 for k, v in big: total += v print(len(big), total, big[99_998], 99_999 in big) nested: {str: [int]} = {} nested["xs"] = [1, 2] nested["xs"].push(3) nested["xs"][0] = 10 print(nested) ``` Output: ```output {"ada": 36, "bob": 26, "cy": 41} 3 true false -1 ada 36 bob 26 cy 41 {"bob": 26, "cy": 41} {"ada": 36, "bob": 26, "cy": 41} ["bob", "cy"] [36, 26, 41] {"the": 3, "cat": 1, "and": 2, "hat": 1, "bat": 1} 0 a 1 b 2 c Store{users: {1: "ada", 2: "bob", 3: "ada"}, counts: {"ada": 2, "bob": 1}} 66666 6666533334 199996 false {"xs": [10, 2, 3]} ``` ## tests/run/json.tn ```tern import json struct Address: city: str zip: str struct User: id: int name: str score: float admin: bool tags: [str] home: Address meta: {str: int} fn main(): u = User(1, "Ada \"the first\"\n", 99.5, true, ["x", "é"], Address("London", "N1"), {"logins": 3}) text = json.encode(u) print(text) v = User(0, "", 0.0, false, [], Address("", ""), {}) ok = json.decode(text, &v) print(ok, v.name == u.name, v.tags, v.home.city, v.meta, v.score) w = User(7, "keep", 1.0, false, ["old"], Address("Paris", "75"), {}) ok2 = json.decode("{\"name\": \"Bo\", \"extra\": [1, {\"a\": null}], \"tags\": [\"a\",\"b\"], \"home\": {\"city\": \"Rome\"}, \"score\": null}", &w) print(ok2, w) bad = json.decode("{\"id\": \"not a number\"}", &w) print(bad, w.id, w.name) xs: [int] = [] print(json.decode("[1, 2, 3.0, -4]", &xs), xs, json.encode(xs)) m: {int: [bool]} = {} print(json.decode("{\"1\": [true, false], \"22\": []}", &m), m, json.encode(m)) print(json.encode("tab\there é"), json.encode(3.25), json.encode([u.home])) s = "" print(json.decode("\"\\u00e9\\ud83d\\ude00 ok\"", &s), s) ``` Output: ```output {"id":1,"name":"Ada \"the first\"\n","score":99.5,"admin":true,"tags":["x","é"],"home":{"city":"London","zip":"N1"},"meta":{"logins":3}} true true ["x", "é"] London {"logins": 3} 99.5 true User{id: 7, name: "Bo", score: 1.0, admin: false, tags: ["a", "b"], home: Address{city: "Rome", zip: "75"}, meta: {}} false 7 Bo true [1, 2, 3, -4] [1,2,3,-4] true {1: [true, false], 22: []} {"1":[true,false],"22":[]} "tab\there é" 3.25 [{"city":"London","zip":"N1"}] true é😀 ok ``` ## tests/run/packages/main.tn ```tern import store from models import User, new_user fn main(): s = store.empty() store.add(&s, " ada ") store.add(&s, "bob") u: User = new_user(9, " cy ") print(store.names(s), u, len(s.users)) ``` Output: ```output ["ada", "bob"] models.User{id: 9, name: "cy"} 2 ``` ## tests/run/packages/models.tn ```tern struct User: id: int name: str fn new_user(id: int, name: str) User: return User(id, _clean(name)) fn _clean(s: str) str: # private to this module return s.trim() ``` ## examples/05_mistakes.tn (rejected on purpose: it shows compile errors) ```tern fn count(xs: take [int]) int: return len(xs) fn bump(n: int): n += 1 # refused: parameters are read-only unless `mut` fn main(): xs = [1, 2, 3] print(count(xs)) # xs is handed over, not copied print(len(xs)) # refused: xs is gone for i in 0..3: i = 10 # refused: loop variables can't change names = [1, 2, 3] for n in names: names.push(n) # refused: changing a list while looping over it ``` ## examples/03_races.tn (rejected on purpose: it shows compile errors) ```tern # Each of these is a data race in Go. Tern refuses to compile them. fn main(): hits = 0 parallel: hits += 1 hits += 1 config = [1, 2, 3] parallel: config.push(4) print(len(config)) total = 0 out = [0] * 100 parallel for i in 0..100: total += i # every iteration writes `total` out[i] = out[i + 1] # reads a neighbour another iteration is writing print(hits, config, total, out) ```