Giving a tiny model a browser, safely

The small model I helped train can't use a web page. It's word salad right now, and handing it the Wikipedia article on Fedora doesn't change that. But the plumbing for live lookups was worth building anyway, because that's the part that has to be right before the model is good enough to matter. And the plumbing has a security problem hiding in it.

The problem

The chat page lives on a home network and has no login. If I let it fetch "any URL the user pastes," then anyone who can reach that page can make the server on the other end request arbitrary addresses. That's server-side request forgery (SSRF): the classic ways to get burned are pointing it at http://localhost:..., at other machines on the LAN, or at a cloud metadata address like 169.254.169.254. The server has network access the visitor doesn't, and I'd be lending it out.

So the design rule is: the fetcher only ever talks to a short list of hosts I've chosen in advance.

The allowlist

ALLOWED_DOMAINS = ["en.wikipedia.org", "wiki.archlinux.org", "wiki.debian.org",
                   "wiki.gentoo.org", "wiki.nobaraproject.org", "docs.fedoraproject.org",
                   "nixos.org", "nebuliteproject.org"]

def allowed(url):
    p = urlparse(url)
    host = (p.hostname or "").lower()
    if p.scheme != "https" or not host:
        return False
    return any(host == d or host.endswith("." + d) for d in ALLOWED_DOMAINS)

The detail that matters is host.endswith("." + d) with the leading dot. Without it, evilwikipedia.org would match wikipedia.org. With it, only real subdomains do.

The rest of the checks

• HTTPS only. No plain http://.
• robots.txt is honoured, using my real User-Agent (a lesson from another post).
• Redirects are followed by hand, and every hop is re-checked. An allowlisted site that redirects to an internal address would otherwise walk straight through the front door.
• A 2 MB cap and a 10 s timeout, so a huge or slow page can't tie the server up.
• The resolved address must be public. After the allowlist passes, I resolve the hostname and refuse anything that lands on a private, loopback or link-local address.

What I actually tested

I fed it evil.com, a plain http:// link, the lookalike en.wikipedia.org.evil.com, localhost, and the metadata IP. All were refused, with a reason. A real Nobara wiki link and a Wikipedia search both fetched fine.

One honest note about that test: every one of those bad inputs was rejected by the allowlist, the first layer. That means the private-address check, the second layer, never actually ran in my tests. It's there, but I haven't seen it fire, and I'd rather say so than count it as verified.

The gap I know about

There's a race between the check and the fetch. I resolve the hostname to test whether it's public, and then the HTTP library resolves it again to connect. A hostile DNS server could answer differently the second time (DNS rebinding). The allowlist makes this hard to exploit here, since an attacker would need to control DNS for one of the allowed domains, so I'm not losing sleep over it. But the fix is to connect to the address I already validated, and I haven't done that yet.

What I'd take from it

Allowlists beat blocklists. It's much easier to say what you're willing to talk to than to enumerate everything you shouldn't. And a security check you've never seen trigger is a hypothesis, not a fact. I'd like to write the test that makes the second layer fire.