haf_system suite: test_blocks_reversible, test_event_after_replay and test_event_massive_sync time out (WitnessNode1 live mode not activated) in the pinned image
Problem and intended result
The haf_system suite of the full slot (#432 (closed)) produced three errors in #436 (closed)'s workflow (436-haf-s-hive-submodule-lacks-hiv-s5e0d19f2-v1, aidev test affected on 2026-11-04, steem-11 under heavy load). The suite ran in its pinned image registry.gitlab.syncad.com/hive/haf/aidev-system-tests@sha256:4e769773…, with hived from develop 46558ea2, independent of #436 (closed)'s change:
test_blocks_reversible.pytest_event_after_replay.pytest_event_massive_sync.py
Each failed with TimeoutError: WitnessNode1: API or live mode not activated on time. 8 tests passed. The ninth failure, test_force_replay_over_populated_database_is_refused, is quarantined for #433 (closed) and #435 (closed).
Before the suite gates candidates, we need to know whether these three are reproducible in the suite's container (no network, uid 1000, HAF_SYSTEM_JOBS=4 nodes in parallel) or depend on host load. A deterministic red would block every candidate that touches hived_api.sql, sql_serializer or haf-local-tools. Do not assume they are transient: find the cause.
Required behavior
- Re-run the three modules in the pinned image as the suite does (
.aidev/run-haf-system.sh --rerun …) on an unloaded host, and again withHAF_SYSTEM_JOBS=1. Record pass/fail and timings, and the node's stderr.log showing why WitnessNode1 did not reach live mode in time. - Fix the cause in the suite's wiring (
.aidev/haf-system.sh/run-haf-system.sh: parallelism, timeouts, HELPY_* retry settings, or the container environment). If the tests themselves are at fault, fix the test. If it is load-only, bound the suite's parallelism so it cannot starve itself, and say so. - Do not quarantine them without a cause.
Acceptance criteria
- The three modules pass in the suite's pinned image through
.aidev/run-haf-system.shat the suite's default parallelism. Or each one is quarantined in.aidev/known_failures.txtwith a root cause stated and a tracking issue. - The result names the cause of the original timeouts.
Verification
Run the suite's command in its pinned image, as #432 (closed)'s verification did (--network none, --user 1000:1000). Gate artifacts of the original run are under steem-11 /var/lib/aidev/gate-artifacts/promote-gates/436-haf-s-hive-submodule-lacks-hiv-s5e0d19f2-v1*, if they are still present.
Integration points
.aidev/haf-system.sh,.aidev/run-haf-system.sh: suite wiring.aidev/known_failures.txt: only with a stated cause
Related-To: #432 (closed) Related-To: #435 (closed)