haf_system suite: test_blocks_reversible, test_event_after_replay and test_event_massive_sync time out (WitnessNode1 live mode not activated) in the pinned image

Problem and intended result

The haf_system suite of the full slot (#432 (closed)) produced three errors in #436 (closed)'s workflow (436-haf-s-hive-submodule-lacks-hiv-s5e0d19f2-v1, aidev test affected on 2026-11-04, steem-11 under heavy load). The suite ran in its pinned image registry.gitlab.syncad.com/hive/haf/aidev-system-tests@sha256:4e769773…, with hived from develop 46558ea2, independent of #436 (closed)'s change:

  • test_blocks_reversible.py
  • test_event_after_replay.py
  • test_event_massive_sync.py

Each failed with TimeoutError: WitnessNode1: API or live mode not activated on time. 8 tests passed. The ninth failure, test_force_replay_over_populated_database_is_refused, is quarantined for #433 (closed) and #435 (closed).

Before the suite gates candidates, we need to know whether these three are reproducible in the suite's container (no network, uid 1000, HAF_SYSTEM_JOBS=4 nodes in parallel) or depend on host load. A deterministic red would block every candidate that touches hived_api.sql, sql_serializer or haf-local-tools. Do not assume they are transient: find the cause.

Required behavior

  • Re-run the three modules in the pinned image as the suite does (.aidev/run-haf-system.sh --rerun …) on an unloaded host, and again with HAF_SYSTEM_JOBS=1. Record pass/fail and timings, and the node's stderr.log showing why WitnessNode1 did not reach live mode in time.
  • Fix the cause in the suite's wiring (.aidev/haf-system.sh / run-haf-system.sh: parallelism, timeouts, HELPY_* retry settings, or the container environment). If the tests themselves are at fault, fix the test. If it is load-only, bound the suite's parallelism so it cannot starve itself, and say so.
  • Do not quarantine them without a cause.

Acceptance criteria

  • The three modules pass in the suite's pinned image through .aidev/run-haf-system.sh at the suite's default parallelism. Or each one is quarantined in .aidev/known_failures.txt with a root cause stated and a tracking issue.
  • The result names the cause of the original timeouts.

Verification

Run the suite's command in its pinned image, as #432 (closed)'s verification did (--network none, --user 1000:1000). Gate artifacts of the original run are under steem-11 /var/lib/aidev/gate-artifacts/promote-gates/436-haf-s-hive-submodule-lacks-hiv-s5e0d19f2-v1*, if they are still present.

Integration points

  • .aidev/haf-system.sh, .aidev/run-haf-system.sh: suite wiring
  • .aidev/known_failures.txt: only with a stated cause

Related-To: #432 (closed) Related-To: #435 (closed)