AIDEV suites miss CI-covered regressions the test stack can reach: paging search, HNSW index, #54 events_id, slice/PCA modes, install lock, stack-pin drift

Problem and intended result

Eric (efrias) reviewed what AIDEV's hivesense suites cover against GitLab CI's sync job, on aidev/integration at 999067d. Several regressions that CI catches can be tested with the AIDEV test stack as it is (.aidev/test-stack.compose.yml: HAF, hivemind and CPU Ollama), but no AIDEV step checks them:

  • The paging search function is never called. hivesense_app.find_nearest_posts_with_embedding (db/search.sql:347) was the function behind #56 (closed). The endpoints call only find_nearest_posts_with_embedding_one_shot, so embed-search's rank checks never reach it. CI's tests/integration/api_node/hivesense_search_test.sh calls it with a small limit, a limit above the corpus and a continuation.
  • The HNSW index is never asserted. Nothing checks that the index named by hivesense_app.get_hnsw_index_name() (db/helpers.sql:139) exists.
  • The #54 (closed) test isn't covered. ensure_context_detached (scripts/sync_embeddings.py:458) must park hafd.contexts.events_id for hivesense_app at hive.unreachable_event_id(). Only CI's events_id_test checks this (.gitlab-ci.yml:376-415).
  • Slice and PCA modes are never installed. reduction_mode, posts_vectors_reduced, the reducing matrix and the slice index have no coverage under AIDEV. #56 (closed) was specific to slice mode.
  • The advisory-lock install path is dark. scripts/install_app.sh:260 re-runs itself through /usr/local/bin/install_with_app_lock.py only when that file exists. The test runtime image doesn't have it, so every AIDEV install takes the WARNING: install_with_app_lock.py wrapper not found fallback, and the skip path (CI's lock_skip_test, .gitlab-ci.yml around line 320-372) is never exercised.
  • Stack pins drift silently. CI picks the newest HAF image for UPSTREAM_BRANCH (.gitlab-ci.yml:168, now 1.28.8-rc3) on every run. The AIDEV stack pins HAF, hafah, reputation_tracker and hivemind by digest, and nothing reports when the two disagree.

Intended result: the full and canary slots close all of these gaps, and the quick slot reports pin drift. What AIDEV still can't reach stays gated by GitLab CI, and .aidev/README.md makes running that pipeline an explicit step before merging into develop.

Required behavior

The two new images are already built and pushed, so this workflow builds nothing. Commit the Dockerfile changes below and pin the digests in the same commit, as the README requires. .aidev/runtime/build.sh derives the tag from a hash of the Dockerfile, so the text must match byte for byte or the hash won't match the pushed tag. After editing, .aidev/runtime/build.sh --tag must print aidev-38c8060c240b6065 and .aidev/runtime/build.sh --image=ollama --tag must print aidev-27681dfd6fb8f272. Those are the tags the digests below were pushed under. If either differs, the Dockerfile text is wrong. Fix the text; don't re-push.

  1. Test runtime image (environment.image in .aidev/project.yaml): registry.gitlab.syncad.com/hive/hivesense/aidev-tests@sha256:087a3f859f6bd4e05e96b4017d7d8235f13d09806e4f09d8acacafad7bf2cfcb. Make these changes to .aidev/runtime/Dockerfile:
    • After the # scripts/hivesense_block_processor.py import, for \unit`` header line, add:
      #   + /usr/local/bin/install_with_app_lock.py from the psql image the hivesense
      #     Dockerfile's PSQL_CLIENT_VERSION names, so `install` takes install_app.sh's
      #     advisory-lock path (it needs only python3 and psycopg2)
    • Between the postgrest and shellcheck FROM lines, add: FROM registry.gitlab.syncad.com/hive/common-ci-configuration/psql:3839feb0c30846aa3ae262143f216a3c1a0fa20b@sha256:6a3794d9fdc4e0590789fc01e48539bcd73273715e3b69dd1f45d3ecde8b1cde AS psql
    • After COPY --from=shellcheck …, add: COPY --from=psql /usr/local/bin/install_with_app_lock.py /usr/local/bin/install_with_app_lock.py
  2. Embedding server image (the ollama anchor in .aidev/test-stack.compose.yml): registry.gitlab.syncad.com/hive/hivesense/aidev-ollama@sha256:933e772d48d20452951670b45903b4ba782deadd55256bcdab98d12458590d88. It serves both all-minilm:l6-v2 and hf.co/mixedbread-ai/mxbai-embed-xsmall-v1:F16 (384 dims, MiniLM tokenizer, so the existing all-MiniLM-L6-v2 tokenizer.json serves it too). Make these changes to .aidev/runtime/ollama.Dockerfile:
    • Replace the header line # copies it there through a volume). Build and pin with .aidev/runtime/build.sh. with:
      # copies it there through a volume). Also CI's slice-mode model,
      # hf.co/mixedbread-ai/mxbai-embed-xsmall-v1:F16 (Matryoshka-trained, 384 dims, MiniLM
      # tokenizer, so the same tokenizer.json serves both). Build and pin with
      # .aidev/runtime/build.sh.
    • Replace ollama pull all-minilm:l6-v2 && ollama list; \ with these two lines:
          ollama pull all-minilm:l6-v2 \
          && ollama pull hf.co/mixedbread-ai/mxbai-embed-xsmall-v1:F16 && ollama list; \
  3. Paging and index checks in embed-search (full mode). Run these against the seeded, embedded corpus in .aidev/run-checks.sh / .aidev/hivesense_smoke.py:
    • find_nearest_posts_with_embedding with _limit 2 returns 2 rows, in the same order the one-shot variant returns them.
    • With _limit 100 it returns every embedded seeded post and nothing else.
    • Continuing with _start_post_id set to the second result returns the remainder: no repeats and no gaps compared with the one-shot order.
    • The index named by get_hnsw_index_name() exists in pg_class.
  4. The #54 (closed) test, done CI's way. After install, set hafd.contexts.events_id = 0 for hivesense_app and call the real sync_embeddings.ensure_context_detached from the tree with psycopg (the runtime image has it). Then assert that events_id = hive.unreachable_event_id(). Mirror CI's failure messages.
  5. Slice mode. Reinstall with --llm=hf.co/mixedbread-ai/mxbai-embed-xsmall-v1:F16 --vector_size=384 --tokenizer-model=all-MiniLM-L6-v2 --use-reduced-embeddings=true --reduction-mode=slice --reduced-dim=256, plus CI's query prefix (.gitlab-ci.yml:441-443) and the other options embed-search already uses. Embed the seeded posts and assert:
    • reduced vectors of the reduced dimension exist for every long seeded post and none for the short one;
    • the slice-mode HNSW index (get_hnsw_index_name()) exists;
    • the paging checks from 3 pass in slice mode (this is the #56 (closed) case);
    • the endpoint rank checks pass. If an assertion that holds for all-minilm doesn't hold for mxbai, weaken that assertion for slice mode only and note why in the README.
  6. PCA mode. Install with --reduction-mode=pca, a small --reduced-dim, and --reduced-matrix-source pointing at a matrix file the step generates. Assert that reduced vectors of that dimension are stored and that search returns the embedded seeded posts.
  7. Lock wrapper. With the wrapper in the image, a normal install must go through it: its log contains no wrapper not found WARNING. Add a skip-path check modelled on CI's lock_skip_test. Hold the app's shared install lock from a background psql session, then run install_app.sh. It must exit 0, print the Skipping hivesense install … lock held by line, and leave the installed schema untouched. Release the lock afterwards, including on failure.
  8. Stack-pin drift step (stackless, in quick, full and canary). It fails when the image pins in .aidev/test-stack.compose.yml don't correspond to .gitlab-ci.yml's UPSTREAM_BRANCH, naming the stale image and the expected branch. One way to do this: carry the tag next to each digest (repo:tag@sha256:…; the release tag for hafah, reputation_tracker and hivemind, and the HAF commit tag plus a recorded branch for HAF), then compare those to UPSTREAM_BRANCH. The step must not need network access.
  9. README. Update .aidev/README.md's step and slot tables, the "Not covered" paragraph and the image sections to match. Add a section saying that GitLab CI remains the gate for what AIDEV can't run: hivesense image builds (build_images), block processing (main_loop.sql scheduler/worker loop, process_blocks.sh, hivesense_block_processor.py), the 1M-block sync, run_pca.py, the rebase rehearsal (legacy_rebase.sql, hivesense_rebase_test.sh), the PostgREST rewriter and the JMeter benchmarks. Before each merge of aidev/integration into develop, run a web- or API-triggered pipeline on aidev/integration (the workflow: rules allow exactly that, .gitlab-ci.yml:57) and merge only when it is green.

Context and decisions

  • Eric proposed items 1-4 and the CI gate in 9. Dan accepted all of them on 2026-10-15. The drift step (8) was proposed by Claude and accepted by Dan in the same exchange.
  • Building images, block processing and the rebase rehearsal need docker-in-docker or a block log. Dan decided they stay in GitLab CI behind the pre-merge pipeline step rather than moving into AIDEV now. Moving image builds into AIDEV is possible later work and is out of scope here.
  • The images were pushed by hand beforehand, as for #61 (closed), so the workflow needs no registry write access. Verified on 2026-10-15:
    • the tests image has the wrapper, and import psycopg2 works;
    • the ollama image lists both models with --network none;
    • mxbai returns 384-dim embeddings offline.
  • Proposed, not confirmed: generate the PCA matrix in the step (for example a deterministic orthonormal projection, or one computed from the full-mode embeddings in plain Python) rather than running run_pca.py. The runtime image has no numpy/sklearn/matplotlib, and four posts would give a degenerate fit anyway. run_pca.py stays CI-only. The worker may choose a different matrix source, provided the image doesn't need rebuilding.
  • Proposed, not confirmed: put slice and PCA in their own steps (for example embed-slice, embed-pca) so their failures are reported separately. Each step ends by reinstalling the default configuration, as embed-search does.

Acceptance criteria

  • With the new digests pinned, every step of the full slot passes on the candidate.
  • If find_nearest_posts_with_embedding's continuation (_start_post_id) skips or repeats a result, or returns more than _limit rows, embed-search fails, naming the paging call. The same breakage in slice mode fails the slice check.
  • If the HNSW index get_hnsw_index_name() names is missing (full or slice mode), the run fails, naming the index.
  • If ensure_context_detached no longer sets events_id to hive.unreachable_event_id(), the run fails with the #54 (closed) message.
  • In slice mode, the seeded long posts have reduced vectors of --reduced-dim dimensions and the short post has none. In PCA mode, the reduced vectors have the configured dimension and search returns the embedded seeded posts.
  • A normal install log contains no install_with_app_lock.py wrapper not found WARNING. While another session holds the app's install lock, install_app.sh exits 0, logs the skip line and leaves the installed schema untouched.
  • If UPSTREAM_BRANCH in .gitlab-ci.yml changes and any of the four stack images isn't re-pinned, the quick slot fails, naming the image. When everything matches, it passes.
  • .aidev/README.md documents the new steps and both image changes, lists what stays CI-only, and states that a green web/API pipeline on aidev/integration is required before merging into develop.

Verification

Run the full slot on the candidate. For each new check, also show that it can fail: break the relevant code in a scratch copy and run the step against it. Examples: make the paging function's continuation skip a row, drop the events_id UPDATE in ensure_context_detached, drop the index, edit UPSTREAM_BRANCH, and remove the wrapper path (the step must fail on the WARNING). Record each failing output in the workflow log. Don't commit the breakages.

Integration points

  • .aidev/runtime/Dockerfile — add the wrapper exactly as above
  • .aidev/runtime/ollama.Dockerfile — pull the slice model exactly as above
  • .aidev/project.yaml — pin environment.image to the new tests digest; slot args for the new steps
  • .aidev/test-stack.compose.yml — pin the new ollama digest; record tags so pin drift can be checked
  • .aidev/run-checks.sh — paging/index, #54 (closed), slice, PCA, lock-skip and pin-drift checks
  • .aidev/hivesense_smoke.py — mode-aware search checks
  • .aidev/README.md — steps, images, CI-only list, pre-merge pipeline rule

Scope boundary

No changes to hivesense's own SQL, scripts, Dockerfile or .gitlab-ci.yml. If a new check finds a real hivesense defect, file a separate issue for it. Don't fix it here, and don't weaken the check to make it pass. Moving image builds or block processing into AIDEV is out of scope.