Integration site: commit a bot gate (fingerprint block, per-IP rate limit, JS cookie challenge) for anonymous page loads
Problem
On 2026-11-13 the integration site (https://denser.discuss.peerverity.info) was rendering about 63 uncached blog pages a second for a distributed crawl: about 22,600 client addresses in 10 minutes, 1–2 requests each, browser user agents (Chrome 116–154 on macOS and Windows), mostly old posts and comments, and robots.txt (Disallow: /, caddy/Caddyfile.routes) ignored. The blog's one Next.js server ran at about 60% CPU, and every post page also called hivesense /similar on api.syncad.com. Real readers waited in line behind it: an uncached AI search took 0.8–5.9 s on the server although hivesense answered in 0.5–2 s, and up to 7 s in a browser.
Two-thirds of the crawl's requests carry Sec-Fetch-Site: none together with Referer: https://denser.discuss.peerverity.info (the bare origin). A real browser never sends a Referer on a typed navigation, so that pair identifies the crawler. The rest look like typed URLs and only differ by not running JavaScript. Per-address rate limits alone do little, because each address sends one or two requests.
What is running now (temporary, not in the repo)
A gate VCL loaded live into denser-integration-varnish-1 with varnishadm vcl.load gate_20261113 + vcl.use. It runs first, then includes the on-disk varnish/default.vcl. Any varnish-apply.sh reload or varnish restart drops it, and the crawl comes back. Results: blog renders went from about 1,900 to 10 per 30 s, blog CPU from about 60% to 0.3%, and a browser loaded trending in 1.6 s instead of 5.3 s and AI search in 2.5 s instead of 7.4 s.
The live gate VCL
vcl 4.1;
# Temporary bot gate (2026-11-13), loaded live with varnishadm; not on disk in the
# checkout. A distributed crawl (residential proxies, browser UAs, robots.txt ignored)
# was rendering ~60 uncached pages/s. Runs before the site's own default.vcl.
import std;
import vsthrottle;
acl gate_lan {
"127.0.0.0"/8;
"10.0.0.0"/8;
"172.16.0.0"/12;
"192.168.0.0"/16;
"::1";
"fc00::"/7;
}
sub bot_gate {
if (req.restarts > 0 || (req.method != "GET" && req.method != "HEAD")) {
return;
}
# Only page documents: API routes, Next's files and the client router's requests pass.
if (req.url ~ "^/[^/?]+/(api|_next)([/?]|$)"
|| req.url ~ "[?&]_rsc([=&]|$)"
|| req.http.RSC
|| req.http.Next-Router-State-Tree
|| req.http.Next-Router-Prefetch
|| req.http.Next-Router-Segment-Prefetch) {
return;
}
set req.http.X-Gate-IP = regsub(req.http.X-Forwarded-For, "^\s*([^,\s]+).*$", "\1");
if (!req.http.X-Gate-Test && std.ip(req.http.X-Gate-IP, "192.0.2.1") ~ gate_lan) {
unset req.http.X-Gate-IP;
return;
}
# A typed navigation (Sec-Fetch-Site: none) never carries a Referer; this crawl
# sends the bare site origin as one.
if (req.http.Sec-Fetch-Site == "none" && req.http.Referer ~ "^https?://[^/]+/?$") {
return (synth(403, "Forbidden"));
}
# 60 page documents a minute per client address; past that, 429 for 5 minutes.
if (vsthrottle.is_denied(req.http.X-Gate-IP, 60, 60s, 300s)) {
return (synth(429, "Too Many Requests"));
}
# Signed-in readers have run the app already; everyone else proves a JS engine once.
if (req.http.Cookie !~ "(^|;)\s*denser_human=1"
&& req.http.Cookie !~ "(^|;)\s*([^=;]*_session(\.[^=;]*)?|observer|account_info)\s*=") {
set req.http.X-Gate = "challenge";
return (synth(200, "OK"));
}
unset req.http.X-Gate-IP;
}
sub vcl_recv {
call bot_gate;
}
sub vcl_synth {
if (req.http.X-Gate-IP && (resp.status == 403 || resp.status == 429 || req.http.X-Gate)) {
set resp.http.Cache-Control = "no-store";
set resp.http.Content-Type = "text/html; charset=utf-8";
if (resp.status == 429) {
set resp.http.Retry-After = "300";
}
if (req.http.X-Gate) {
synthetic({"<!doctype html><html><head><meta name="robots" content="noindex"><title>Loading…</title><script>document.cookie="denser_human=1; Max-Age=2592000; Path=/; Secure; SameSite=Lax";location.reload();</script></head><body><noscript>This site needs JavaScript.</noscript></body></html>"});
} else {
synthetic(resp.reason);
}
return (deliver);
}
}
include "/etc/varnish/default.vcl";
Required behavior
Make the gate part of the integration stack, so it survives restarts, upgrades and varnish-apply.sh. It applies only to page documents (GET/HEAD), not to /<app>/api/*, /<app>/_next/*, _rsc or Next router requests (RSC, Next-Router-* headers), or /robots.txt and /status:
- Fingerprint block:
Sec-Fetch-Site: noneplus a Referer that is a bare origin → 403,Cache-Control: no-store. - Per-address rate limit: 60 page documents a minute per client address (the first
X-Forwarded-Forentry, which caddy sets from the PROXY header) → 429 withRetry-After: 300for 5 minutes. - JS challenge: an anonymous request without a
denser_human=1cookie (and without the signed-in cookies the VCL already recognises) gets a tinyno-store,noindexHTML page. The page setsdenser_human=1(30 days,Path=/,Secure,SameSite=Lax) and reloads. The cookie must not change caching: the HTML cache still keys anonymous pages on URL and locale only. - Exemptions: private and LAN addresses (
10/8,172.16/12,192.168/16,127/8,fc00::/7,::1) are exempt, so the fleet's tests, live checks and curl probes see real pages. A request header (the live VCL usesX-Gate-Test: 1) opts an exempt client back into the gate, for testing. - Wallet: apply it to the wallet's pages too. Today the wallet bypasses varnish (
caddy -> wallet:80). The crawl currently hits only/blog, so this part is lower priority, but it should not be left as an open door. - Without the HTML cache: the gate must also work when
DENSER_HTML_CACHEis off. Either put it in caddy (fingerprint and challenge with matchers, and a rate limit if the caddy image has a module for it), or make it its own small varnish stage that runs even without the cache. Pick the simpler of the two. If the gate is varnish-only, say so in the README. - Session stacks (
Caddyfile.session,session-stack.sh) are not affected unless that comes for free. They are not public. - Document it in
stack/integration/README.md(what the gate does, how to turn it off with an env knob, e.g.DENSER_BOT_GATE=off, and how to test it), and cover all three rules plus the exemptions intest-cache.shortest-routes.sh.
Acceptance
- On the integration site after deploy: a curl from outside the LAN gets the challenge page; with
denser_human=1it gets the real page; with the crawler header pair it gets 403. A LAN curl gets the real page, and a LAN curl withX-Gate-Test: 1gets the challenge. - A headless browser loads trending, a post, AI search and a profile with one challenge reload on its first page.
- The blog's request log shows tens, not thousands, of page renders a minute while the crawl continues. On deploy, the live
gate_20261113VCL is replaced by the committed one.