## Summary During the 2026-07-09 network-wide P2P disruption (see hive!2064 for the p2p mechanism), a mainnet HAF 1.28.5 node's write-ahead-log processor hit a foreign-key violation in `hive.set_irreversible()`, causing hived to shut down (cleanly, by design). No hived restart was involved — the node was live the whole time — so this is the same broken invariant as #223 (closed 2024-07 via !511) reached through a **different path** than the ctrl-c/restart scenario that fix addressed. ## The failure ``` 2026-07-09T19:23:05.102621 livesync_data_dumper.cpp:287 run ] Write-ahead log processor detected SQL error: ERROR: insert or update on table "hive_state" violates foreign key constraint "fk_1_hive_irreversible_data" DETAIL: Key (consistent_block)=(107991362) is not present in table "blocks". CONTEXT: SQL statement "UPDATE hafd.hive_state SET consistent_block = _block_num" PL/pgSQL function hive.set_irreversible(integer) line 36 at SQL statement 2026-07-09T19:23:05.102677 livesync_data_dumper.cpp:248 boost_se_body ] Exiting hived->postgresql write-ahead log processing thread 2026-07-09T19:23:05.152388 application.cpp:425 wait ] Attempting to request for application shutdown... ``` hived then shut down in an orderly fashion ("exited cleanly", exit status 0). ## Conditions leading up to it The node lived through the P2P disconnect storm (network-wide, ~18:54–19:23 UTC): peer count repeatedly collapsed to 0, with fork switches between the majority chain and a minority branch happening across the network. The last minute of its log: - `19:22:37` — `peer_count changed: 0` (node cut off during the storm's second wave) - `19:23:04` — got a peer back; began sync-fetching missed blocks - `19:23:05.039–.050` — dumped blocks 107991367–107991372 (`block_num_rendezvous_trigger "Dump whole block N"`), block stats show `"type":"sync"` and **LIB jumping 107991360 → 107991365 within ~10 ms** as fast-confirms for the missed span arrived - `19:23:05.102` — `set_irreversible(107991362)` fails: block 107991362 not in `blocks` Note the visible dumps in this window start at 107991367 — there are no `Dump whole block` lines for 107991361–366 in the excerpt we have (which begins 19:22:37; they may have been dumped earlier, possibly from the minority fork). ## Hypothesis During the storm the node switched forks at least once. Block 107991362 was plausibly dumped as a reversible block from the minority branch, removed (or superseded) during fork-switch cleanup, and the majority-chain replacement had not yet reached the `blocks` table when the fast-confirm-driven LIB jump triggered `set_irreversible(107991362+)`. I.e. an ordering race between reversible-block replacement after a fork switch and irreversibility advancement, exposed only because the storm produced a fork switch immediately followed by a multi-block LIB jump during live sync. #223's reproduction required a hived restart to make the block vanish from `block_num_rendezvous_trigger`; here the same "irreversibility marker advanced past a block missing from `blocks`" state was reached with no restart, so the !511 fix doesn't cover it. ## Environment - HAF 1.28.5 (hived + sql_serializer, live sync), mainnet - PostgreSQL 17 - Reported by @mahdiyari; a second HAF 1.28.5 node of his survived the same event, consistent with a timing-sensitive race ## Logs / data that would help (requested from the reporter) 1. **Full hived console log from 18:50 UTC through the shutdown (~19:23:10)** — the excerpt we have starts at 19:22:37. Specifically wanted: any `switch_forks` / `pop_block` lines, the `Dump whole block` lines for 107991355–107991366 (do they appear once or twice, and when?), and `peer_count` transitions across both storm waves (~18:54–19:11 and ~19:17–19:23). 2. **Whether the node restarted successfully afterward**, and the first ~200 lines of that startup log (did HAF's consistency recovery repair the state, and from which block?). 3. **Postgres-side state**, if queried before/after restart (adjust names to the 1.28.5 schema if needed): - `SELECT * FROM hafd.hive_state;` - `SELECT num, encode(hash,'hex') FROM hafd.blocks WHERE num BETWEEN 107991355 AND 107991375 ORDER BY num;` — is there a gap at 107991362? - `SELECT num, encode(hash,'hex'), fork_id FROM hafd.blocks_reversible WHERE num BETWEEN 107991355 AND 107991375 ORDER BY num, fork_id;` — was 107991362 present with a different fork id? - `SELECT * FROM hafd.fork ORDER BY id DESC LIMIT 10;` — fork events registered during the storm window 4. **p2p.log for 18:50–19:25**, if the p2p appender was enabled — to correlate his node's disconnect/reconnect timing with the fork switches. Item 1 is the most valuable: it should show directly whether 107991362 was dumped from the losing fork and cleaned up without a replacement dump.