alpha2: hived-compatible P2P + byte-identical mainnet sync
Phase 2: live network. ghive-sync downloads blocks from real mainnet
hived seed peers (seed.hive.blog, seed.openhive.network, etc.), fully
validates each one (signature + merkle + every op evaluator), and writes
them to a local block_log byte-identical to hived's canonical output.
Verified against the in-tree ./block_log: the first 2,000 mainnet blocks
produced by ghive-sync are byte-identical to ./block_log[0:241810] per
'cmp'. No verify-against mismatches across the run.
P2P stack (hived wire-compatible):
- stcp_socket-style ECDHE-secured TCP transport (AES-256-CBC, key derived
from sha256(SHA512(ECDH-X)), IV via half-swap of CityHash128). Full
envelope + magicless-zstd matches hived bit-for-bit.
- Dual hello (legacy_hello_message 5006 + new hello_message 5018) so
older mainnet peers accept us; HelloMessage carries signed shared
secret proving ECDHE owner + chain_id + node_id in user_data.
- AddressRequest/AddressMessage exchange + persistent peer DB
(<data-dir>/peers.json) with sticky seeds + exponential dial backoff.
- INV exchange + catch-up sync state machine
(FetchBlockchainItemIDs → BlockchainItemIDsInventory → FetchItems →
BlockMessage), single-goroutine apply with reorder buffer.
- Per-peer + total outstanding-fetch back-pressure (hysteresis 75/25%)
prevents apply-queue saturation under multi-peer ingest.
Witness production:
- Real WIF parser (5J… → 32-byte secret).
- secp256k1 signing of SHA256(unsigned_block_header) on every produced
block; recovered-pubkey verify against on-chain Witness.BlockSigningKey
on every inbound block.
Testnet (5+ nodes, 21 deterministic init witnesses, distinct chain_id):
- scripts/testnet-up N + testnet-config emits per-node config.ini with
round-robin witness assignment + p2p ports.
- Smoke (3-node + minimal seeds): full mesh in ~10 seconds; chain
advances every 3s; all heads converge.
block_log writer (internal/blocklog/writer.go):
- Per-block zstd compression with hived's exact parameters
(ZSTD_f_zstd1_magicless, contentSize=0, dictID=0, checksum=0, level=15
matching programs/util/compress_block_log default).
- 10 trained zstd dictionaries embedded; dict selection
min(block_num/1_000_000, 9). Verified byte-identical against the
cross-dict-boundary blocks (999_998..1_000_005).
Other fixes:
- verify_authority no longer rejects 0-signature txs (pow_operation has
empty required-authority by design — its proof of work IS the auth).
- ghive-sync recent-block-cache enabled so the apply hook can stream
raw block bytes into the writer.
Replay regression: 8.22M-block mainnet block_log clears strict replay
with 0 errors; perf 55-77K bps depending on phase.
Tooling:
- cmd/ghive-sync: --target-block, --verify-against, --compress-blocks,
--no-default-seeds, --extra-seed, --show-seeds, --p2p-endpoint.
- scripts/testnet-{up,down,status,heads}: orchestrate local testnet.
- GHIVE_P2P_TRACE=1: per-peer wire trace.
41 commits since alpha1 across 9 agent branches (gossip / sync /
discovery / mainnet-skeleton / interop-debug / sigauth / applyq /
zstd / signing / testnet / p2p-msg / p2p-conn).
Known follow-ups: block 25,502 active-auth on Transfer needs sig
recovery debugging; gossip+sync apply-loop concurrent-map race when
both delegates run on the same chain.Database (testnet smoke flagged
this; out of scope for the mainnet-sync task).