Tags

Tags give the ability to mark specific points in history as being important

  • alpha5

    659cba76 · ghive-sync: --log-adaptive-batch CLI flag plumbs into sync.Config · 
    alpha5: 6-agent perf wave (pipeline + compressed-blocks + pebble + peers + profile + mem-default)

16 commits across 6 parallel worktrees. Highlights ranked by impact:

PIPELINE (#1+#2): Parallel within-block tx-sig recovery on the per-peer
receive goroutine, plus a tx-sig cache (8K-entry LRU) to dedupe
duplicate deliveries from multiple peers. +44% bps avg (855→1193,
920→1494 best). Tried separate decode/sig worker pool + singleflight
on signerCache — both regressed because back-pressure stalled the TCP
receive loop and triggered peer drops.

COMPRESSED BLOCKS (#3+#8): proto v106→v107 + advertise zstd dict
catalogue in hello user_data so peers send compressed_block_message
(5002) instead of block_message (1001). 100% inbound traffic switched
(0 → 55,972 compressed deliveries in 60s). Decompression via existing
internal/blocklog/zstd. Lazy-decompress cache in chain/recent_blocks.

PEERS (#6+#7): max-outbound 8→32, max-concurrent-dials →24. Fixed
LegacyAddressMessage (5010) wire-encoding bug — fc::ip::address packs
as LE uint32 — that was causing peerdb to never fill past the 8 seeds.
Per-peer adaptive FetchBatch sized to recent blocks/sec, bounded
[50, 5000]. Result: 8→22 peers verified in mainnet runs (2.6× fan-out).

PEBBLE (#4): --cache-mb=1024 (was 256MB hard-coded). SoftCommit +
chain-driven N-block batched commits + auto-replay crash recovery
(restart compares persisted DGPO head vs block_log head, replays gap).
Per-batch fsync via --commit-batch-fsync. Honest finding: per the
profile, secp256k1 is 88% of CPU and pebble fsync is 0.6%, so the
batching framework is for crash safety not throughput.

PROFILE (#10): Full pprof report at docs/perf/sync-profile.md.
Top hotspot: secp256k1 RecoverPublicKey 88.4% cum. Top alloc:
state.Session.Put overlay copies 168MB / 200K blocks. Recommendations
for the next round (skip-sig on trusted self-replay → 5-7×; per-tx
parallel sig recovery — already in pipeline agent's work). Small wins:
state.Range backend-only fast path (3% gain). pprof flag wiring.

MEM DEFAULT for --verify-against (separate 1-line commit before
fan-out). Verify-against runs throw away state — pebble fsync wasted.
Now defaults to mem; explicit --backend=X overrides.

State-commit batching (#9, agent-commit-batch) was developed in
parallel but skipped at merge as duplicate-concept with pebble's
already-merged batching framework. Worktree preserved if we want
to revisit the deeper Database.BatchSize design.

All ./internal/p2p/... + ./internal/chain/ + ./cmd/ghive-sync/ tests
green. Replay regression: 2M blocks, 0 errors. Bottleneck has moved
off network/peers and into per-block apply CPU; next round of work
should target sig recovery itself (decred/secp256k1 has FieldVal as
35-30-10% leaves of ECDSA recovery).

  • alpha4

    bd287ae8 · p2p/sync: per-block-ID signer cache + lock-free head hint · 
    alpha4: 12× sync throughput (100 → 1,179 bps mainnet catch-up)

Five sync-perf wins after profiling vs real mainnet seeds:

1. ce587ef — Quick wins bundle: log every 1000 (was per-block stderr),
   --backend=mem flag for ghive-sync, MaxBlocksPerInv 200→1000,
   FetchBatch tuning, INV-driven catch-up rate calculation.

2. d4b57c8 — Parallel block-signature recovery across peer goroutines.
   secp256k1 RecoverPublicKey was the dominant CPU cost (~47% of
   samples). With N peers we now get N-way parallel recovery for
   free; applyLoop never blocks on sig math.

3. 969b704 — Throttle inventory broadcast during catch-up. Stops
   spamming INV when we're miles behind head; saves bandwidth + CPU.

4. f7aa93a — Catch-up kicker keeps multiple chunks in flight per peer.
   Removed the len(missing)>0 gate on walk-forward AND fan out to all
   peers (not just the responder) so the slowest peer doesn't bottleneck
   the round.

5. b58e1ab — Per-block-ID signer cache (8K-entry LRU) + lock-free
   headBlockHint atomic mirror. Catch-up redundancy across N peers
   meant the same block triggered N sig recoveries; cache dedupes
   them. headBlockHint avoids the applyMu.RLock + state.Session
   materialize per inbound block.

Measured 60s mainnet sync (8 default seeds, --verify-against=./block_log):
- baseline (alpha3): 100 bps
- after all 5 wins: 1,179 bps avg, 5,042 bps peak burst

Time to 9M projection: 25h → ~2.1h.

All sig + block + state correctness preserved (verify-against 0
mismatches, replay 2M blocks 0 errors, all p2p+chain+blocklog tests
green under -race).

5 commits since alpha3.

  • alpha3

    612945d5 · chain: serialize ApplyBlock + read accessors against witness goroutine · 
    alpha3: race-free apply path + account_authority correctness

Fixes the two known-issues from alpha2's commit message:

- gossip + sync apply-loop race: gossip's apply goroutine deleted; sync
  owns the SINGLE apply path. Re-broadcast happens via
  sync.Config.OnBlockApplied -> gossip.BroadcastAppliedBlock (fires
  inside sync's applyLoop, same goroutine that committed the block).
- chain.Database.ApplyBlock now under applyMu (RWMutex). Read accessors
  HeadBlock/HeadBlockID/ScheduledWitness/GetDGPO take RLock so the
  witness loop can poll them without racing the applier. Witness reads
  dgpo via p.Chain.GetDGPO() instead of state.NewSession(p.State).
- fcraw.InternString locked with sync.RWMutex (the original comment
  said this was needed for production, and Phase 2 made it true).

Sig recovery:
- pow_apply now creates account_authority alongside Account (mirrors
  hived's hive_evaluator.cpp:424-430 / :561-567). Without this, every
  Transfer signed by an early-mainnet PoW-registered account fails
  verify_authority on the live-sync path.
- Genesis seeds the legacy 'steem' system account's authority with
  HIVE_STEEM_PUBLIC_KEY_STR (was empty). Mirrors database_init.cpp
  :276-291.

Result: ghive-sync against real mainnet seeds clears past block
906,000 byte-identical to the local block_log. Block 906,012 trips
the pre-existing iter15-era witness-schedule cascade — orthogonal to
this work.

Replay regression: 0 errors @ 50K+ bps. All p2p + chain + witness +
fcraw test suites green under go test -race. Testnet smoke (3-node)
converges with zero race-detector firings (pre-fix: node-1 died with
10 race reports).

8 commits since alpha2.

  • alpha2

    3caced5b · p2p/sync: regression test for outstanding-cap throttle path · 
    alpha2: hived-compatible P2P + byte-identical mainnet sync

Phase 2: live network. ghive-sync downloads blocks from real mainnet
hived seed peers (seed.hive.blog, seed.openhive.network, etc.), fully
validates each one (signature + merkle + every op evaluator), and writes
them to a local block_log byte-identical to hived's canonical output.

Verified against the in-tree ./block_log: the first 2,000 mainnet blocks
produced by ghive-sync are byte-identical to ./block_log[0:241810] per
'cmp'. No verify-against mismatches across the run.

P2P stack (hived wire-compatible):
- stcp_socket-style ECDHE-secured TCP transport (AES-256-CBC, key derived
  from sha256(SHA512(ECDH-X)), IV via half-swap of CityHash128). Full
  envelope + magicless-zstd matches hived bit-for-bit.
- Dual hello (legacy_hello_message 5006 + new hello_message 5018) so
  older mainnet peers accept us; HelloMessage carries signed shared
  secret proving ECDHE owner + chain_id + node_id in user_data.
- AddressRequest/AddressMessage exchange + persistent peer DB
  (<data-dir>/peers.json) with sticky seeds + exponential dial backoff.
- INV exchange + catch-up sync state machine
  (FetchBlockchainItemIDs → BlockchainItemIDsInventory → FetchItems →
   BlockMessage), single-goroutine apply with reorder buffer.
- Per-peer + total outstanding-fetch back-pressure (hysteresis 75/25%)
  prevents apply-queue saturation under multi-peer ingest.

Witness production:
- Real WIF parser (5J… → 32-byte secret).
- secp256k1 signing of SHA256(unsigned_block_header) on every produced
  block; recovered-pubkey verify against on-chain Witness.BlockSigningKey
  on every inbound block.

Testnet (5+ nodes, 21 deterministic init witnesses, distinct chain_id):
- scripts/testnet-up N + testnet-config emits per-node config.ini with
  round-robin witness assignment + p2p ports.
- Smoke (3-node + minimal seeds): full mesh in ~10 seconds; chain
  advances every 3s; all heads converge.

block_log writer (internal/blocklog/writer.go):
- Per-block zstd compression with hived's exact parameters
  (ZSTD_f_zstd1_magicless, contentSize=0, dictID=0, checksum=0, level=15
  matching programs/util/compress_block_log default).
- 10 trained zstd dictionaries embedded; dict selection
  min(block_num/1_000_000, 9). Verified byte-identical against the
  cross-dict-boundary blocks (999_998..1_000_005).

Other fixes:
- verify_authority no longer rejects 0-signature txs (pow_operation has
  empty required-authority by design — its proof of work IS the auth).
- ghive-sync recent-block-cache enabled so the apply hook can stream
  raw block bytes into the writer.

Replay regression: 8.22M-block mainnet block_log clears strict replay
with 0 errors; perf 55-77K bps depending on phase.

Tooling:
- cmd/ghive-sync: --target-block, --verify-against, --compress-blocks,
  --no-default-seeds, --extra-seed, --show-seeds, --p2p-endpoint.
- scripts/testnet-{up,down,status,heads}: orchestrate local testnet.
- GHIVE_P2P_TRACE=1: per-peer wire trace.

41 commits since alpha1 across 9 agent branches (gossip / sync /
discovery / mainnet-skeleton / interop-debug / sigauth / applyq /
zstd / signing / testnet / p2p-msg / p2p-conn).

Known follow-ups: block 25,502 active-auth on Transfer needs sig
recovery debugging; gossip+sync apply-loop concurrent-map race when
both delegates run on the same chain.Database (testnet smoke flagged
this; out of scope for the mainnet-sync task).

  • alpha1

    3f96f1b6 · cashout: hbd_print_rate=0 means 0% to HBD, not unset · 
    alpha1: strict replay clears full 8.22M-block local block_log with 0 errors

Major fixes:
- iter17: block-1 off-by-99 in CurrentAslot (closes the 5.21M cascade
  that included gxt-1080-sc-0028 disable, median feed at 5,210,400,
  martin-stuessy 91-sat shortfall).
- HBDPrintRate=0 was treated as 'unset' in payCreateHbd and fell back
  to 10000 (100% HBD); hived treats 0 as the post-HF14 0%-cap. Closes
  the cryptos 35,850-sat shortfall at block 6,198,475.

Performance:
- Genesis → 1M: ~103K blocks/s
- Cliff (3M → 4M): 10.2K blocks/s (was 2.0K pre-perf-cliff = 5.1×)
- 5M → 8M steady-state: 11–12K blocks/s
- Snapshot/Restore now O(1) via chainbase-style undo log instead of
  O(overlay-size) shallow copy (the cliff root cause).
- Multiple alloc reductions (commentKey 1-alloc, memIter zero-copy,
  WitnessStore + AccountStore object pooling, fcraw short-string
  interning, Session.Put buffer reuse).
- Native uint128 in funds reward math (replaces math/big.Int).
- WitnessStore Mutate + dirtyOrder + btree path hints.

Tooling:
- ghive-tools replay: --snapshot-every / --snapshot-dir periodic dir
  snapshots; --load-snapshot (file or dir auto-detect);
  --save-dir-snapshot-at / --save-dir-snapshot-to for exact-block dir
  snapshots; --cpuprofile / --memprofile.
- cmd/drift-hunter: per-block diff vs hived API across DGPO,
  witness_schedule, feed_history, reward_funds, per-witness top-N,
  per-account (--account=NAME).
- scripts/hived-at <BLOCK> dgpo|witness|account|schedule|raw|shell|stop:
  one-shot wrapper around the hived checkpoint framework, reuses
  prepared dirs, per-block port allocation, daemonized via setsid.

Replay status: 0 errors through end of local block_log (8,221,695).
Next divergence (if any) requires a longer block_log to find.