FK violation in hive.set_irreversible during live sync after P2P disconnect storm — recurrence of #223 via a restart-free path

Summary

During the 2026-07-09 network-wide P2P disruption (see hive!2064 (merged) for the p2p mechanism), a mainnet HAF 1.28.5 node's write-ahead-log processor hit a foreign-key violation in hive.set_irreversible(), causing hived to shut down (cleanly, by design). No hived restart was involved — the node was live the whole time — so this is the same broken invariant as #223 (closed) (closed 2024-07 via !511 (merged)) reached through a different path than the ctrl-c/restart scenario that fix addressed.

The failure

2026-07-09T19:23:05.102621 livesync_data_dumper.cpp:287  run ] Write-ahead log processor detected SQL error: ERROR:  insert or update on table "hive_state" violates foreign key constraint "fk_1_hive_irreversible_data"
DETAIL:  Key (consistent_block)=(107991362) is not present in table "blocks".
CONTEXT:  SQL statement "UPDATE hafd.hive_state SET consistent_block = _block_num"
PL/pgSQL function hive.set_irreversible(integer) line 36 at SQL statement

2026-07-09T19:23:05.102677 livesync_data_dumper.cpp:248  boost_se_body ] Exiting hived->postgresql write-ahead log processing thread
2026-07-09T19:23:05.152388 application.cpp:425           wait ] Attempting to request for application shutdown...

hived then shut down in an orderly fashion ("exited cleanly", exit status 0).

Conditions leading up to it

The node lived through the P2P disconnect storm (network-wide, ~18:54–19:23 UTC): peer count repeatedly collapsed to 0, with fork switches between the majority chain and a minority branch happening across the network. The last minute of its log:

  • 19:22:37 — peer_count changed: 0 (node cut off during the storm's second wave)
  • 19:23:04 — got a peer back; began sync-fetching missed blocks
  • 19:23:05.039–.050 — dumped blocks 107991367–107991372 (block_num_rendezvous_trigger "Dump whole block N"), block stats show "type":"sync" and LIB jumping 107991360 → 107991365 within ~10 ms as fast-confirms for the missed span arrived
  • 19:23:05.102 — set_irreversible(107991362) fails: block 107991362 not in blocks

Note the visible dumps in this window start at 107991367 — there are no Dump whole block lines for 107991361–366 in the excerpt we have (which begins 19:22:37; they may have been dumped earlier, possibly from the minority fork).

Hypothesis

During the storm the node switched forks at least once. Block 107991362 was plausibly dumped as a reversible block from the minority branch, removed (or superseded) during fork-switch cleanup, and the majority-chain replacement had not yet reached the blocks table when the fast-confirm-driven LIB jump triggered set_irreversible(107991362+). I.e. an ordering race between reversible-block replacement after a fork switch and irreversibility advancement, exposed only because the storm produced a fork switch immediately followed by a multi-block LIB jump during live sync.

#223 (closed)'s reproduction required a hived restart to make the block vanish from block_num_rendezvous_trigger; here the same "irreversibility marker advanced past a block missing from blocks" state was reached with no restart, so the !511 (merged) fix doesn't cover it.

Environment

  • HAF 1.28.5 (hived + sql_serializer, live sync), mainnet
  • PostgreSQL 17
  • Reported by @mahdiyari; a second HAF 1.28.5 node of his survived the same event, consistent with a timing-sensitive race

Logs / data that would help (requested from the reporter)

  1. Full hived console log from 18:50 UTC through the shutdown (~19:23:10) — the excerpt we have starts at 19:22:37. Specifically wanted: any switch_forks / pop_block lines, the Dump whole block lines for 107991355–107991366 (do they appear once or twice, and when?), and peer_count transitions across both storm waves (~18:54–19:11 and ~19:17–19:23).
  2. Whether the node restarted successfully afterward, and the first ~200 lines of that startup log (did HAF's consistency recovery repair the state, and from which block?).
  3. Postgres-side state, if queried before/after restart (adjust names to the 1.28.5 schema if needed):
    • SELECT * FROM hafd.hive_state;
    • SELECT num, encode(hash,'hex') FROM hafd.blocks WHERE num BETWEEN 107991355 AND 107991375 ORDER BY num; — is there a gap at 107991362?
    • SELECT num, encode(hash,'hex'), fork_id FROM hafd.blocks_reversible WHERE num BETWEEN 107991355 AND 107991375 ORDER BY num, fork_id; — was 107991362 present with a different fork id?
    • SELECT * FROM hafd.fork ORDER BY id DESC LIMIT 10; — fork events registered during the storm window
  4. p2p.log for 18:50–19:25, if the p2p appender was enabled — to correlate his node's disconnect/reconnect timing with the fork switches.

Item 1 is the most valuable: it should show directly whether 107991362 was dumped from the losing fork and cleaned up without a replacement dump.