url
The url module parses, builds and resolves URLs and escapes their parts, by the rules of Go's net/url (RFC 3986). A Url is its text, written the way format writes it, plus where each part is: you read the parts with methods (u.host(), u.path()), decoded where that applies (the path, user and fragment), and change them with setters. The query stays as written until you ask for its pairs.
import url
fn main():
u = url.parse("https://ada@example.com:8443/docs/a%20b?q=tern&tag=x&tag=y#intro").unwrap()
print(u.scheme(), u.user(), u.host(), u.port(), u.path(), u.raw_query(), u.fragment())
q = u.query()
print(q.get("q"), q.get_all("tag"), q.has("page"))
print(url.format(u))
print(url.join("https://example.com/docs/guide/", "../api?v=2"))https ada example.com 8443 /docs/a b q=tern&tag=x&tag=y intro
tern ["x", "y"] false
https://ada@example.com:8443/docs/a%20b?q=tern&tag=x&tag=y#intro
Ok("https://example.com/docs/api?v=2")Functions
fn parse(text: str) Result[Url, str] a URL, absolute or relative
fn format(u: Url) str u as text, escaped where needed
fn build(p: Parts) Url a Url from its parts (escaping them)
fn parts(u: Url) Parts every part, decoded, to change and build again
fn set_path(u: mut Url, path: str) and set_scheme, set_user, set_password, set_host,
set_port (int), set_raw_query, set_fragment
fn resolve(base: Url, ref: Url) Url ref resolved against base (RFC 3986 section 5)
fn join(base: str, ref: str) Result[str, str] the same for text: a link found on a page
fn host_port(u: Url) str "example.com:8443", "[::1]:80", "example.com"
fn request_uri(u: Url) str an HTTP request line's target: "/docs/a%20b?q=tern"
fn escaped_path(u: Url) str the path as the text has it
fn escaped_fragment(u: Url) str the fragment as the text has it
fn query(u: Url) Query the query's pairs (badly escaped ones are skipped)
fn parse_query(raw: str) Result[Query, str] "a=1&b=x+y" as pairs
fn new_query() Query
fn get(q: Query, key: str) str the first value ("" if absent)
fn get_all(q: Query, key: str) [str] every value, in order
fn has(q: Query, key: str) bool
fn add(q: mut Query, key: str, value: str) another pair
fn set(q: mut Query, key: str, value: str) the only pair with this key
fn remove(q: mut Query, key: str)
fn encode_query(q: Query) str the pairs as a query, in order
fn query_string(pairs: {str: str}) str a map as a query, keys sorted
fn query_escape(s: str) str for a query key or value: space is "+"
fn query_unescape(s: str) Result[str, str]
fn path_escape(s: str) str for one path segment: "/" is escaped too
fn path_unescape(s: str) Result[str, str]The parts
| Method | https://ada:pw@example.com:8443/a%20b?q=1#top |
|---|---|
u.scheme() | "https" (lower case when parsed) |
u.user(), u.password() | "ada", "pw" (decoded) |
u.host() | "example.com" (an IPv6 address without its brackets: "::1") |
u.port() | 8443 (0 when the URL has none) |
u.path() | "/a b" (decoded); u.escaped_path() is "/a%20b" |
u.raw_query() | "q=1" (as written; u.query() decodes it) |
u.fragment() | "top" (decoded) |
u.opaque() | what follows scheme: when it is neither // nor a path: mailto:ada@x.org has "ada@x.org" |
u.raw_path() and u.raw_fragment() give those parts as the text has them when that isn't how escaping the decoded part would write them (/a%2Fb decodes to /a/b, but must stay %2F), else ""; u.force_query() and u.omit_host() tell a lone ? and scheme:/path without //. format(parse(s)) is s for any URL Go's url.Parse accepts, apart from these: a port above 65535, a : in a host outside brackets, and text before a [ in the host are errors here; a scheme comes out in lower case, an empty or zero port (x:/, x:0), user (@x) or password (u:@x) is dropped, a port loses its leading zeros, and what must be escaped is (/a b becomes /a%20b).
To change a part, call its setter; format escapes what needs it:
import url
fn main():
u = url.parse("https://api.example.com").unwrap()
u.set_path("/v1/search results")
q = url.new_query()
q.add("q", "tern & go")
q.add("page", "2")
u.set_raw_query(url.encode_query(q))
print(url.format(u))https://api.example.com/v1/search%20results?q=tern+%26+go&page=2Each setter writes the text again. To change several parts at once, or to make a URL with no text to start from, fill in a url.Parts (fields scheme opaque user password host port path raw_path raw_query force_query fragment raw_fragment omit_host) and build it:
import url
fn main():
p = url.parse("http://example.com/a?x=1").unwrap().parts()
p.scheme = "https"
p.port = 8443
p.path = "/b c"
print(url.format(url.build(p)))
print(url.format(url.build(url.Parts("http", "", "", "", "::1", 8080, "/", "", "", false, "", "", false))))https://example.com:8443/b%20c?x=1
http://[::1]:8080/Print a Url with url.format(u) (or u.parts()): print(u) shows its text and positions.
Errors
import url
fn main():
for s in ["http://x/%zz", "http://x:99999/", "http://[::1", "http://x\ny", "1http://x"]:
print(url.parse(s))Err("url: invalid escape "%zz" in "http://x/%zz"")
Err("url: invalid port "99999" in "http://x:99999/"")
Err("url: missing "]" in host in "http://[::1"")
Err("url: invalid control character in "http://x\ny"")
Err("url: first path segment cannot contain a colon in "1http://x"")Resolving links
resolve and join follow RFC 3986 section 5, including its dot segments:
import url
fn main():
base = "http://a/b/c/d;p?q"
for r in ["g", "../g", "../../../g", "?y", "#s", "//g", "g;x=1/../y"]:
print(r, url.join(base, r).unwrap_or(""))g http://a/b/c/g
../g http://a/b/g
../../../g http://a/g
?y http://a/b/c/d;p?y
#s http://a/b/c/d;p?q#s
//g http://g
g;x=1/../y http://a/b/c/yQueries
A Query keeps its pairs in the order given (q.keys, q.vals), and a key may repeat. parse_query is an Err for a bad escape or a ; (once a separator, and servers still disagree on it), while u.query() skips such pairs, as a server reading a request would:
import url
fn main():
print(url.parse_query("tag=a&tag=b&name=Ada+Lovelace&empty="))
print(url.parse_query("a=1;b=2"))
print(url.query_string({"q": "a&b", "lang": "en"}))Ok(url.Query{keys: ["tag", "tag", "name", "empty"], vals: ["a", "b", "Ada Lovelace", ""]})
Err("url: invalid ";" in query (use "&" between pairs)")
lang=en&q=a%26bSpeed and memory
bench/std does the same work with Go's net/url and Rust's url crate (WHATWG URLs, which also normalizes) on a million generated URLs with paths, queries, some escapes, ports, user info and fragments. For parse each program also reads the host, path, query and port.
| ns per URL | Tern | Go | Rust |
|---|---|---|---|
parse | 177 | 322 | 237 |
format | 4 | 165 | 23 |
resolve + format | 95 | 538 | 115 |
query(), every pair | 79 | 195 | 100 |
query_escape + unescape | 90 | 199 | 241 |
| peak memory (1M parsed) | 238 MB | 529 MB | 496 MB |
(format is a string Tern already has, shared; Rust's to_string copies its text.)
A Url is its text and five ints of positions, like Rust's. Parsing a URL already written the way format writes it (the scheme in lower case, nothing that must be escaped or dropped) finds the positions in one pass and keeps the string it was given, so a parsed URL costs 48 bytes over its text and format hands back that text. Any other URL is parsed part by part and written once. Resolving a link with no scheme or host joins the base's text up to its path, the merged path (one pass), and the link's query and fragment. Escaping scans 16 bytes at a time with a table per kind of part; a part with nothing to change comes back without a copy. The answers agree with Go's on 200,000 random URL-like strings, for every part, format, resolve, both escapes and unescapes, and parse_query, except for the differences listed above.