Production guard: post/comment counts vs hivemind (#63)

Closes #63 (closed). This is its second fix item; the README half was !123 (merged).

Problem

Two count errors reached production, and both were found by hand:

• #42 (closed): content volume was exactly 2x too high;
• #45 (closed): account post counts were about 19% too high.

The guards added afterwards run only on the 2016 CI replay, which cannot reach 2020–2025 data. Nothing checked these numbers where they were actually wrong.

Change

tests/parity/content_vs_hivemind.sql checks haf_stats against hivemind on any node where hivemind is co-located. hive_posts keeps deleted content as rows, so both sides count creations.

What	Rule
Posts, per day and per author	exact: matched on every one of 3,864 days, 2016-03-24 to 2026-10-21
Comments, per day	haf_stats ≥ hivemind, and at most 2% above it
Comments, per author	≥ hivemind, and no further above it than the whole network is on the same days

Why comments have a band: hivemind's massive sync drops a re-created reply whose parent was also deleted and re-created in the same batch. That is filed as hivemind#394 (closed). On 52 days of 2016–2021 hivemind is short by up to 182 comments (1.33% of that day), and haf_stats matches hived's post_count there. A count below hivemind's always fails.

What it samples:

• Days: every complete day of the last 365, plus one day a month back to 2016-04. The day of the month is hashed from the run date.
• Authors: 200 over the last 30 days, plus 20 per monthly day (the #45 (closed) class lived in old account rows). The sample is hashed from the run date, and authors are matched by name.

When it can't run, it says so:

• A NOT RUN warning (exit 0) when hivemind is not live or an index is missing or invalid.
• An error when hivemind or hafd.contexts is unreadable (wrong role), a column is missing, a sampled author is unknown to HAF, or either the recent or the monthly half compared nothing.
• A quiet skip only when hivemind_app is absent.

Running it:

• Read-only, bounded by both context heads and hivemind's last_completed_block_num, with statement_timeout 5 min and lock_timeout 5 s.
• The README gives the standalone docker exec … haf_admin command and a cron snippet that captures stderr.
• On production, run that standalone file only, not scripts/run_verification.sh. Guard 8 there only skips, since there is no hivemind.

CI: tests/ci_content_oracle.sh runs the real guard in the stub lane against a closed-world fixture (tests/content_oracle_fixture.sql). Each case runs inside a transaction that is rolled back. There are 16 cases: a baseline, 11 failure cases (one per rule, plus wrong role / RLS / unmapped author / nothing compared) and 2 NOT RUN cases. The fixture includes a midnight-edge block and differing hivemind and HAF account ids.

Evidence (testapi, 1.28.8 full-history node, 2026-10-23, read-only)

• Calibration: the full history, day by day, gave the rules above.
• The guard: "480 days and 2,484 account checks agree with hivemind … through 2026-10-22", in 68 s, with 2 in-band days reported (2020-02-09 +10, 2021-04-09 +45).
• Negative controls on real data all fail:
  • network counts ×2 (#42 (closed)): 480 mismatches;
  • account posts ×1.19 on old rows (#45 (closed)): 8 mismatches;
  • one day's comments −1: 1 mismatch.

Review

3 independent reviewers per round (guard logic, CI harness, docs/operations), 2 rounds.

• Round 1: led to the comment band, NOT RUN handling, completion-marker bound, wider sampling and this CI harness.
• Round 2: the harness reviewer predicted the wrong-role case would fail on a schema USAGE error, and pipeline 183088 confirmed it. Fixed by checking schema USAGE first. Other round-2 fixes: the per-author bound, a separately counted monthly half, unmapped authors, hashed sampling, and docs (stderr capture, read locks, operational vs data errors).