Tern

url

The url module parses, builds and resolves URLs and escapes their parts, by the rules of Go's net/url (RFC 3986). A Url is its text, written the way format writes it, plus where each part is: you read the parts with methods (u.host(), u.path()), decoded where that applies (the path, user and fragment), and change them with setters. The query stays as written until you ask for its pairs.

import url

fn main():
    u = url.parse("https://ada@example.com:8443/docs/a%20b?q=tern&tag=x&tag=y#intro").unwrap()
    print(u.scheme(), u.user(), u.host(), u.port(), u.path(), u.raw_query(), u.fragment())
    q = u.query()
    print(q.get("q"), q.get_all("tag"), q.has("page"))
    print(url.format(u))
    print(url.join("https://example.com/docs/guide/", "../api?v=2"))
Output
https ada example.com 8443 /docs/a b q=tern&tag=x&tag=y intro
tern ["x", "y"] false
https://ada@example.com:8443/docs/a%20b?q=tern&tag=x&tag=y#intro
Ok("https://example.com/docs/api?v=2")

Functions

fn parse(text: str) Result[Url, str]          a URL, absolute or relative
fn format(u: Url) str                         u as text, escaped where needed
fn build(p: Parts) Url                        a Url from its parts (escaping them)
fn parts(u: Url) Parts                        every part, decoded, to change and build again
fn set_path(u: mut Url, path: str)            and set_scheme, set_user, set_password, set_host,
                                              set_port (int), set_raw_query, set_fragment
fn resolve(base: Url, ref: Url) Url           ref resolved against base (RFC 3986 section 5)
fn join(base: str, ref: str) Result[str, str] the same for text: a link found on a page
fn host_port(u: Url) str                      "example.com:8443", "[::1]:80", "example.com"
fn request_uri(u: Url) str                    an HTTP request line's target: "/docs/a%20b?q=tern"
fn escaped_path(u: Url) str                   the path as the text has it
fn escaped_fragment(u: Url) str               the fragment as the text has it
fn query(u: Url) Query                        the query's pairs (badly escaped ones are skipped)

fn parse_query(raw: str) Result[Query, str]   "a=1&b=x+y" as pairs
fn new_query() Query
fn get(q: Query, key: str) str                the first value ("" if absent)
fn get_all(q: Query, key: str) [str]          every value, in order
fn has(q: Query, key: str) bool
fn add(q: mut Query, key: str, value: str)    another pair
fn set(q: mut Query, key: str, value: str)    the only pair with this key
fn remove(q: mut Query, key: str)
fn encode_query(q: Query) str                 the pairs as a query, in order
fn query_string(pairs: {str: str}) str        a map as a query, keys sorted

fn query_escape(s: str) str                   for a query key or value: space is "+"
fn query_unescape(s: str) Result[str, str]
fn path_escape(s: str) str                    for one path segment: "/" is escaped too
fn path_unescape(s: str) Result[str, str]

The parts

Methodhttps://ada:pw@example.com:8443/a%20b?q=1#top
u.scheme()"https" (lower case when parsed)
u.user(), u.password()"ada", "pw" (decoded)
u.host()"example.com" (an IPv6 address without its brackets: "::1")
u.port()8443 (0 when the URL has none)
u.path()"/a b" (decoded); u.escaped_path() is "/a%20b"
u.raw_query()"q=1" (as written; u.query() decodes it)
u.fragment()"top" (decoded)
u.opaque()what follows scheme: when it is neither // nor a path: mailto:ada@x.org has "ada@x.org"

u.raw_path() and u.raw_fragment() give those parts as the text has them when that isn't how escaping the decoded part would write them (/a%2Fb decodes to /a/b, but must stay %2F), else ""; u.force_query() and u.omit_host() tell a lone ? and scheme:/path without //. format(parse(s)) is s for any URL Go's url.Parse accepts, apart from these: a port above 65535, a : in a host outside brackets, and text before a [ in the host are errors here; a scheme comes out in lower case, an empty or zero port (x:/, x:0), user (@x) or password (u:@x) is dropped, a port loses its leading zeros, and what must be escaped is (/a b becomes /a%20b).

To change a part, call its setter; format escapes what needs it:

import url

fn main():
    u = url.parse("https://api.example.com").unwrap()
    u.set_path("/v1/search results")
    q = url.new_query()
    q.add("q", "tern & go")
    q.add("page", "2")
    u.set_raw_query(url.encode_query(q))
    print(url.format(u))
Output
https://api.example.com/v1/search%20results?q=tern+%26+go&page=2

Each setter writes the text again. To change several parts at once, or to make a URL with no text to start from, fill in a url.Parts (fields scheme opaque user password host port path raw_path raw_query force_query fragment raw_fragment omit_host) and build it:

import url

fn main():
    p = url.parse("http://example.com/a?x=1").unwrap().parts()
    p.scheme = "https"
    p.port = 8443
    p.path = "/b c"
    print(url.format(url.build(p)))
    print(url.format(url.build(url.Parts("http", "", "", "", "::1", 8080, "/", "", "", false, "", "", false))))
Output
https://example.com:8443/b%20c?x=1
http://[::1]:8080/

Print a Url with url.format(u) (or u.parts()): print(u) shows its text and positions.

Errors

import url

fn main():
    for s in ["http://x/%zz", "http://x:99999/", "http://[::1", "http://x\ny", "1http://x"]:
        print(url.parse(s))
Output
Err("url: invalid escape "%zz" in "http://x/%zz"")
Err("url: invalid port "99999" in "http://x:99999/"")
Err("url: missing "]" in host in "http://[::1"")
Err("url: invalid control character in "http://x\ny"")
Err("url: first path segment cannot contain a colon in "1http://x"")

resolve and join follow RFC 3986 section 5, including its dot segments:

import url

fn main():
    base = "http://a/b/c/d;p?q"
    for r in ["g", "../g", "../../../g", "?y", "#s", "//g", "g;x=1/../y"]:
        print(r, url.join(base, r).unwrap_or(""))
Output
g http://a/b/c/g
../g http://a/b/g
../../../g http://a/g
?y http://a/b/c/d;p?y
#s http://a/b/c/d;p?q#s
//g http://g
g;x=1/../y http://a/b/c/y

Queries

A Query keeps its pairs in the order given (q.keys, q.vals), and a key may repeat. parse_query is an Err for a bad escape or a ; (once a separator, and servers still disagree on it), while u.query() skips such pairs, as a server reading a request would:

import url

fn main():
    print(url.parse_query("tag=a&tag=b&name=Ada+Lovelace&empty="))
    print(url.parse_query("a=1;b=2"))
    print(url.query_string({"q": "a&b", "lang": "en"}))
Output
Ok(url.Query{keys: ["tag", "tag", "name", "empty"], vals: ["a", "b", "Ada Lovelace", ""]})
Err("url: invalid ";" in query (use "&" between pairs)")
lang=en&q=a%26b

Speed and memory

bench/std does the same work with Go's net/url and Rust's url crate (WHATWG URLs, which also normalizes) on a million generated URLs with paths, queries, some escapes, ports, user info and fragments. For parse each program also reads the host, path, query and port.

ns per URLTernGoRust
parse177322237
format416523
resolve + format95538115
query(), every pair79195100
query_escape + unescape90199241
peak memory (1M parsed)238 MB529 MB496 MB

(format is a string Tern already has, shared; Rust's to_string copies its text.)

A Url is its text and five ints of positions, like Rust's. Parsing a URL already written the way format writes it (the scheme in lower case, nothing that must be escaped or dropped) finds the positions in one pass and keeps the string it was given, so a parsed URL costs 48 bytes over its text and format hands back that text. Any other URL is parsed part by part and written once. Resolving a link with no scheme or host joins the base's text up to its path, the merged path (one pass), and the link's query and fragment. Escaping scans 16 bytes at a time with a table per kind of part; a part with nothing to change comes back without a copy. The answers agree with Go's on 200,000 random URL-like strings, for every part, format, resolve, both escapes and unescapes, and parse_query, except for the differences listed above.

Source: docs/std/url.md. Every example on this page is compiled and run by tern x test docs.