Integration site: commit a bot gate (fingerprint block, per-IP rate limit, JS cookie challenge) for anonymous page loads

Problem

On 2026-11-13 the integration site (https://denser.discuss.peerverity.info) was rendering about 63 uncached blog pages a second for a distributed crawl: about 22,600 client addresses in 10 minutes, 1–2 requests each, browser user agents (Chrome 116–154 on macOS and Windows), mostly old posts and comments, and robots.txt (Disallow: /, caddy/Caddyfile.routes) ignored. The blog's one Next.js server ran at about 60% CPU, and every post page also called hivesense /similar on api.syncad.com. Real readers waited in line behind it: an uncached AI search took 0.8–5.9 s on the server although hivesense answered in 0.5–2 s, and up to 7 s in a browser.

Two-thirds of the crawl's requests carry Sec-Fetch-Site: none together with Referer: https://denser.discuss.peerverity.info (the bare origin). A real browser never sends a Referer on a typed navigation, so that pair identifies the crawler. The rest look like typed URLs and only differ by not running JavaScript. Per-address rate limits alone do little, because each address sends one or two requests.

What is running now (temporary, not in the repo)

A gate VCL loaded live into denser-integration-varnish-1 with varnishadm vcl.load gate_20261113 + vcl.use. It runs first, then includes the on-disk varnish/default.vcl. Any varnish-apply.sh reload or varnish restart drops it, and the crawl comes back. Results: blog renders went from about 1,900 to 10 per 30 s, blog CPU from about 60% to 0.3%, and a browser loaded trending in 1.6 s instead of 5.3 s and AI search in 2.5 s instead of 7.4 s.

The live gate VCL
vcl 4.1;

# Temporary bot gate (2026-11-13), loaded live with varnishadm; not on disk in the
# checkout. A distributed crawl (residential proxies, browser UAs, robots.txt ignored)
# was rendering ~60 uncached pages/s. Runs before the site's own default.vcl.

import std;
import vsthrottle;

acl gate_lan {
	"127.0.0.0"/8;
	"10.0.0.0"/8;
	"172.16.0.0"/12;
	"192.168.0.0"/16;
	"::1";
	"fc00::"/7;
}

sub bot_gate {
	if (req.restarts > 0 || (req.method != "GET" && req.method != "HEAD")) {
		return;
	}
	# Only page documents: API routes, Next's files and the client router's requests pass.
	if (req.url ~ "^/[^/?]+/(api|_next)([/?]|$)"
	    || req.url ~ "[?&]_rsc([=&]|$)"
	    || req.http.RSC
	    || req.http.Next-Router-State-Tree
	    || req.http.Next-Router-Prefetch
	    || req.http.Next-Router-Segment-Prefetch) {
		return;
	}
	set req.http.X-Gate-IP = regsub(req.http.X-Forwarded-For, "^\s*([^,\s]+).*$", "\1");
	if (!req.http.X-Gate-Test && std.ip(req.http.X-Gate-IP, "192.0.2.1") ~ gate_lan) {
		unset req.http.X-Gate-IP;
		return;
	}
	# A typed navigation (Sec-Fetch-Site: none) never carries a Referer; this crawl
	# sends the bare site origin as one.
	if (req.http.Sec-Fetch-Site == "none" && req.http.Referer ~ "^https?://[^/]+/?$") {
		return (synth(403, "Forbidden"));
	}
	# 60 page documents a minute per client address; past that, 429 for 5 minutes.
	if (vsthrottle.is_denied(req.http.X-Gate-IP, 60, 60s, 300s)) {
		return (synth(429, "Too Many Requests"));
	}
	# Signed-in readers have run the app already; everyone else proves a JS engine once.
	if (req.http.Cookie !~ "(^|;)\s*denser_human=1"
	    && req.http.Cookie !~ "(^|;)\s*([^=;]*_session(\.[^=;]*)?|observer|account_info)\s*=") {
		set req.http.X-Gate = "challenge";
		return (synth(200, "OK"));
	}
	unset req.http.X-Gate-IP;
}

sub vcl_recv {
	call bot_gate;
}

sub vcl_synth {
	if (req.http.X-Gate-IP && (resp.status == 403 || resp.status == 429 || req.http.X-Gate)) {
		set resp.http.Cache-Control = "no-store";
		set resp.http.Content-Type = "text/html; charset=utf-8";
		if (resp.status == 429) {
			set resp.http.Retry-After = "300";
		}
		if (req.http.X-Gate) {
			synthetic({"<!doctype html><html><head><meta name="robots" content="noindex"><title>Loading…</title><script>document.cookie="denser_human=1; Max-Age=2592000; Path=/; Secure; SameSite=Lax";location.reload();</script></head><body><noscript>This site needs JavaScript.</noscript></body></html>"});
		} else {
			synthetic(resp.reason);
		}
		return (deliver);
	}
}

include "/etc/varnish/default.vcl";

Required behavior

Make the gate part of the integration stack, so it survives restarts, upgrades and varnish-apply.sh. It applies only to page documents (GET/HEAD), not to /<app>/api/*, /<app>/_next/*, _rsc or Next router requests (RSC, Next-Router-* headers), or /robots.txt and /status:

  1. Fingerprint block: Sec-Fetch-Site: none plus a Referer that is a bare origin → 403, Cache-Control: no-store.
  2. Per-address rate limit: 60 page documents a minute per client address (the first X-Forwarded-For entry, which caddy sets from the PROXY header) → 429 with Retry-After: 300 for 5 minutes.
  3. JS challenge: an anonymous request without a denser_human=1 cookie (and without the signed-in cookies the VCL already recognises) gets a tiny no-store, noindex HTML page. The page sets denser_human=1 (30 days, Path=/, Secure, SameSite=Lax) and reloads. The cookie must not change caching: the HTML cache still keys anonymous pages on URL and locale only.
  4. Exemptions: private and LAN addresses (10/8, 172.16/12, 192.168/16, 127/8, fc00::/7, ::1) are exempt, so the fleet's tests, live checks and curl probes see real pages. A request header (the live VCL uses X-Gate-Test: 1) opts an exempt client back into the gate, for testing.
  5. Wallet: apply it to the wallet's pages too. Today the wallet bypasses varnish (caddy -> wallet:80). The crawl currently hits only /blog, so this part is lower priority, but it should not be left as an open door.
  6. Without the HTML cache: the gate must also work when DENSER_HTML_CACHE is off. Either put it in caddy (fingerprint and challenge with matchers, and a rate limit if the caddy image has a module for it), or make it its own small varnish stage that runs even without the cache. Pick the simpler of the two. If the gate is varnish-only, say so in the README.
  7. Session stacks (Caddyfile.session, session-stack.sh) are not affected unless that comes for free. They are not public.
  8. Document it in stack/integration/README.md (what the gate does, how to turn it off with an env knob, e.g. DENSER_BOT_GATE=off, and how to test it), and cover all three rules plus the exemptions in test-cache.sh or test-routes.sh.

Acceptance

  • On the integration site after deploy: a curl from outside the LAN gets the challenge page; with denser_human=1 it gets the real page; with the crawler header pair it gets 403. A LAN curl gets the real page, and a LAN curl with X-Gate-Test: 1 gets the challenge.
  • A headless browser loads trending, a post, AI search and a profile with one challenge reload on its first page.
  • The blog's request log shows tens, not thousands, of page renders a minute while the crawl continues. On deploy, the live gate_20261113 VCL is replaced by the committed one.