uri

Imported from benpate/uri (AGENTS.md). Install upstream with npx skills add benpate/uri. Copyright stays with the author.

uri — Notes for AI Agents

• IsPublicIP (public.go) and IsLocalHostname (localhost.go) differ by INPUT, not by range. For a literal IP they are exact complements: IsLocalHostname parses the string and delegates to NotPublicIP, so blockedRanges in public.go is the single source of truth and the two can no longer drift. FuzzSSRFSafety pins that bidirectionally. They still are not interchangeable, because IsPublicIP takes a resolved net.IP while IsLocalHostname takes a string that may be a name — and a name is only classified by its text, never resolved. For an SSRF check, always use IsPublicIP on the resolved IP at connection time (e.g. in a net.Dialer.Control hook); a hostname string check cannot catch a name that resolves to a private address. benpate/remote is that caller: publicIPs resolves the host and dials only addresses IsPublicIP accepts.
• IsLocalHostname classifies names by text, and the text list is broad. Beyond localhost, 127.0.0.1 and ::1 it covers the RFC 6761 .localhost TLD (any *.localhost), FQDN trailing-dot forms, the /etc/hosts aliases (localhost.localdomain, ip6-localhost, ip6-loopback, loopback), and the .local, .internal, and host.docker.internal suffixes. It also calls NormalizeHost first, so it accepts a full URL, host:port, userinfo, and bracketed IPv6 without any pre-stripping at the call site.
• The cloud-metadata endpoint 169.254.169.254 is deliberately caught by the link-local case in both IsLocalHostname and IsPublicIP. It looks like an ordinary public-ish address but routes to instance credentials — the single most important SSRF target. Don't "simplify" the link-local branches away.
• IsPublicIP works on the resolved net.IP, not on a string — and that's the point. A string like 0x7f.1 or 2130706433 is loopback once parsed, but a naive string check won't see it. Resolve first, classify second.
• ValidateHostname exempts IP addresses and local names before applying DNS rules. 127.0.0.1 and friday.local are valid hostnames here even though they have no IANA TLD. Only after those exemptions does it enforce RFC 1035 label/length limits and require the final segment to be a real IANA TLD. If you change the ordering, you'll start rejecting valid loopback/local inputs.
• The TLD list is embedded at build time and loaded once in init(). RefreshTLDs can re-fetch the live list from IANA at runtime, but it is best-effort and fails closed onto the list already in memory. Three guards make that true, and all three must stay: a non-200 status is rejected (an IANA 503 page would otherwise be parsed as the TLD list), the download is capped with io.LimitReader(…, 1<<20), and a response that parses to zero TLDs is rejected rather than published. That last one is why parsing (parseTLDs) is split from publishing (importTLDs) — publishing an empty map makes every public hostname fail validation process-wide, so RefreshTLDs inspects the parse result before the atomic swap.
• Refresh _iana.txt from https://data.iana.org/TLD/tlds-alpha-by-domain.txt whenever you touch this package. Emissary calls RefreshTLDs only once per node at boot, so a node uses this embedded copy until that fetch lands, and for its whole life if IANA is unreachable then. A stale copy rejects every host under a newly delegated TLD with a 422 that reads as bad input. init loads it without the empty-list guard, so run the tests afterward: TestTLD fails on a botched download.
• Host returns an RFC 6454 origin, not a raw authority. Per §6.2 it omits the port when it is the scheme's default, so https://server.com:443 and https://server.com both serialize to https://server.com — and IsSameOrigin, which is built on Host, treats them as one origin. The trim is scheme-specific (:80 for http, :443 for https only), and safe against IPv6 because url.URL.Host always brackets an IPv6 literal, putting the port at the tail. Federated peers do emit explicit default ports; without this they would read as cross-origin.
• SafeURL and IsSafeRedirectURL are one policy with two return shapes — both call parseSafeURL, which is the ONLY copy of the accept/reject rule. IsSafeRedirectURL reports it; SafeURL re-serializes the parsed value on top (percent-encoding attribute-breaking characters, plus the ' that url.URL.String leaves alone). Never re-implement the check in either one. IsValidRedirectURL is the third caller and the strictest. The single permitted gap: SafeURL's "" doubles as its rejection sentinel, so a safe value that re-serializes to "" (only the empty string, now that // is refused outright) is indistinguishable from a rejection — harmless, since both are a blank href. FuzzSafeURLMatchesRedirectGuard pins all of this, including that anything IsValidRedirectURL accepts is also accepted by IsSafeRedirectURL.
• Protocol returns a usable prefix or nothing at all. It returns "" unless the value genuinely had a scheme://. Two shapes are rejected that url.Parse will happily hand you a scheme for: no scheme at all (which used to yield a bare "://"), and an opaque URI, where Opaque != "" means there was no // — example.com:8080/path parses as scheme example.com, and mailto:a@b.c as mailto. Note Scheme does NOT share this rule: it still returns localhost for localhost:3000, so Protocol != Scheme + "://" in exactly those cases.
• Paired Is… / Not… predicates are intentional, not redundant. NotLocalURL, NotPublicIP, NotValidTLD, etc. exist so callers read naturally at the call site (if uri.NotPublicIP(ip)). Each Not… is a one-line negation of its Is… twin — keep them in sync.
• Every rejection from ParseURL, ValidateURL, and ValidateHostname is a 422. derp.Wrap inherits the inner error's code, and a non-derp error (*url.Error, an idna error) reads as 500, so a bare wrap would answer bad user input with a server error. Each wrap of a third-party error passes derp.WithCode(http.StatusUnprocessableEntity); TestValidateURL_RejectionsAreValidationErrors pins it.
• PrependProtocol tells host:port from a scheme by the port. url.Parse reads localhost:8080 as the scheme localhost with an opaque payload, the same way it reads mailto:me@example.com. hasOpaqueScheme counts a payload that starts with a digits-only port as host:port, and anything else as a scheme of its own, which returns "". The protocol is guessed from the whole value, because IsLocalHostname normalizes every shape, while url.Parse(uri).Hostname() is empty for the schemeless input this function exists to handle. FuzzPrependProtocolWellFormed pins that the added protocol is always GuessProtocolForHostname's.
• Hostname lower-cases and strips, it does not validate. It will happily return garbage from garbage. Run the result through ValidateHostname if the input is untrusted.

A redirect target is judged the way a BROWSER parses it, not the way net/url does

parseSafeURL is what IsSafeRedirectURL, SafeURL, and IsValidRedirectURL all ask, and it has to agree with the WHATWG URL parser, because a browser is what finally follows the value. Two divergences from net/url were live open-redirect bypasses, and both are now closed.

• A leading run of slashes opens an AUTHORITY, and a backslash is a slash. For an http(s) URL a browser reads \ as /, and it enters authority parsing as soon as the first two characters are any mix of the two, ignoring every slash after that. So /\evil.com, \\evil.com, \/evil.com, /\/evil.com and ///evil.com all resolve to https://evil.com, while net/url reports an empty Host and a same-site path for every one of them. hasAuthorityPrefix therefore tests the first two characters and nothing else: a backslash later in the path (/foo/\bar) stays on this host, and rejecting it would break ordinary paths.
• Leading whitespace is stripped before parsing. A browser removes C0 controls and spaces from both ends first, so " //evil.com" is off-site to a browser and a harmless relative path to net/url. parseSafeURL trims the same set before it parses. Interior tabs and newlines, which a browser also removes, still fail closed the old way, because url.Parse rejects the control byte.
• A browser deletes tabs and newlines from ANYWHERE in a URL, and Path works on the DECODED path. So /%09/evil.com reaches safeLocalPath as /\t/evil.com, gets past the leading-slash collapse, and then reads as //evil.com. Emissary puts the sign-in next through PathAndQuery into Hx-Redirect, so this is a live post-login open redirect unless safeLocalPath deletes tabOrNewline before it collapses. The fuzz oracle isProtocolRelative deletes the same set; without it the fuzzer cannot see this class at all.

authoritySlashes is the single copy of that character set, shared with safeLocalPath in path.go, which collapses exactly the same run for PathAndQuery. Keep both on the constant. TestIsSafeRedirectURL_AuthorityPrefixBypass pins every spelling above, and its twin pins the paths that must stay safe.

What did NOT change: an absolute off-site http(s) URL is still safe. All of this is about values that only LOOK same-site. One behavior did change deliberately: // is now refused rather than accepted as a blank href.

IsValidRedirectURL is the rule for a destination chosen by someone else, and its NAME is load-bearing

IsSafeRedirectURL accepts a relative path, because most callers redirect inside their own server. IsValidRedirectURL refuses one, because its callers forward to a URL named by a remote server, which has no business naming a path here — a hostile home server answering an Activity Intent could otherwise bounce a visitor into /signout. It is IsSafeRedirectURL plus "absolute, with a valid host". ValidateHostname exempts loopback, literal IPs, and .local, so a home server in local development still passes; the TLD rule means a made-up TLD does not, and neither does .example, so write tests against example.com.

The name matches CodeQL's barrier-guard pattern, and almost nothing else here does. The go/unvalidated-url-redirection query recognises a sanitizer only by callee name, matched against (?i)(is_?)?(local_?url|valid_?redir(ect)?)(ur[li])?. IsValidRedirectURL matches. IsSafeRedirectURL, IsValidURL, and NotValidRedirectURL do not. So a call site that has to satisfy the scanner must be written in the positive form, as if !uri.IsValidRedirectURL(target), and the Not… twin — which exists only for the package's paired-predicate convention — will not be credited. Renaming this function silently re-opens every alert it closes.

Known defects, reproduced and not yet fixed

• Path and PathAndQuery return the DECODED path, so a percent-encoded delimiter comes back live. Both use parsedURL.Path where they want EscapedPath(). PathAndQuery("/foo%3Fbar") returns /foo?bar, turning an encoded ? into a real query delimiter, and %23 into a fragment; raw spaces and raw UTF-8 come through too, in a value the doc comment calls suitable for an HTTP Location header. Blast radius of the fix is one pinned test. The decoded tab or newline that made this an open redirect is deleted in safeLocalPath (see above); what remains is a changed meaning, never an off-site hop.
• ValidateHostname accepts a full URL when, and only when, it names a local host. IsValidHostname("http://localhost/../etc") is TRUE while IsValidHostname("https://example.com") is FALSE, because the IsLocalHostname early-out normalizes a URL down to a bare host and the public-domain path below it does not. Acceptance therefore depends on which half you land in, and the local half is the security-sensitive one. ParseURL is unaffected — it passes an already-bare parsed.Hostname().
• ValidateHostname accepts control and invisible characters (BUG-171). It converts with idna.ToASCII, the Punycode profile, which maps and validates nothing, so "\u0080.com", "a\u200d.com" (a zero-width joiner), and a soft hyphen all pass. idna.Lookup.ToASCII rejects them, but also changes what some inputs normalize to.
• IsValidTLD re-implements ValidateTLD instead of calling it (BUG-172). Both copies look the TLD up as given against lower-cased keys, so IsValidTLD("COM") is FALSE; ValidateHostname lower-cases first, so nothing in the package notices.
• Scheme and Protocol disagree on opaque URLs. Scheme("http:8080") is "http" but Protocol("http:8080") is "". Protocol has an explicit Opaque != "" rule and Scheme has none, though the README presents them as a pair.
• Do not re-flag these as new: IsValidIP4Address("::ffff:1.2.3.4") == false is a deliberate text-form classification and drives no security decision here; trimDefaultPort's :80 trim cannot eat example.com:8080; the init() panic on embed failure is programmer-error-only; and RefreshTLDs taking no context.Context is a deliberate fire-and-forget with a 30s client timeout.
• Hostname is a thin alias for NormalizeHost — keep it that way. It used to have its own cut-at-the-first-delimiter logic, which left the querystring attached (example.com?q=1), left userinfo attached (evil.com@good.com), and truncated a bracketed IPv6 literal to "[". Callers build federated @user@host handles and referer checks out of this, so both functions must answer identically for every shape; TestHostname_MatchesNormalizeHost pins that. Note that an IPv6 result comes back unbracketed (::1), so it can contain colons.