AIDEV suites miss CI-covered regressions the test stack can reach: paging search, HNSW index, #54 events_id, slice/PCA modes, install lock, stack-pin drift
Problem and intended result
Eric (efrias) reviewed what AIDEV's hivesense suites cover against GitLab CI's sync job, on aidev/integration at 999067d. Several regressions that CI catches can be tested with the AIDEV test stack as it is (.aidev/test-stack.compose.yml: HAF, hivemind and CPU Ollama), but no AIDEV step checks them:
- The paging search function is never called.
hivesense_app.find_nearest_posts_with_embedding(db/search.sql:347) was the function behind #56 (closed). The endpoints call onlyfind_nearest_posts_with_embedding_one_shot, soembed-search's rank checks never reach it. CI'stests/integration/api_node/hivesense_search_test.shcalls it with a small limit, a limit above the corpus and a continuation. - The HNSW index is never asserted. Nothing checks that the index named by
hivesense_app.get_hnsw_index_name()(db/helpers.sql:139) exists. - The #54 (closed) test isn't covered.
ensure_context_detached(scripts/sync_embeddings.py:458) must parkhafd.contexts.events_idforhivesense_appathive.unreachable_event_id(). Only CI'sevents_id_testchecks this (.gitlab-ci.yml:376-415). - Slice and PCA modes are never installed.
reduction_mode,posts_vectors_reduced, the reducing matrix and the slice index have no coverage under AIDEV. #56 (closed) was specific to slice mode. - The advisory-lock install path is dark.
scripts/install_app.sh:260re-runs itself through/usr/local/bin/install_with_app_lock.pyonly when that file exists. The test runtime image doesn't have it, so every AIDEV install takes theWARNING: install_with_app_lock.py wrapper not foundfallback, and the skip path (CI'slock_skip_test,.gitlab-ci.ymlaround line 320-372) is never exercised. - Stack pins drift silently. CI picks the newest HAF image for
UPSTREAM_BRANCH(.gitlab-ci.yml:168, now1.28.8-rc3) on every run. The AIDEV stack pins HAF, hafah, reputation_tracker and hivemind by digest, and nothing reports when the two disagree.
Intended result: the full and canary slots close all of these gaps, and the quick slot reports pin drift. What AIDEV still can't reach stays gated by GitLab CI, and .aidev/README.md makes running that pipeline an explicit step before merging into develop.
Required behavior
The two new images are already built and pushed, so this workflow builds nothing. Commit the Dockerfile changes below and pin the digests in the same commit, as the README requires. .aidev/runtime/build.sh derives the tag from a hash of the Dockerfile, so the text must match byte for byte or the hash won't match the pushed tag. After editing, .aidev/runtime/build.sh --tag must print aidev-38c8060c240b6065 and .aidev/runtime/build.sh --image=ollama --tag must print aidev-27681dfd6fb8f272. Those are the tags the digests below were pushed under. If either differs, the Dockerfile text is wrong. Fix the text; don't re-push.
- Test runtime image (
environment.imagein.aidev/project.yaml):registry.gitlab.syncad.com/hive/hivesense/aidev-tests@sha256:087a3f859f6bd4e05e96b4017d7d8235f13d09806e4f09d8acacafad7bf2cfcb. Make these changes to.aidev/runtime/Dockerfile:- After the
# scripts/hivesense_block_processor.py import, for \unit`` header line, add:# + /usr/local/bin/install_with_app_lock.py from the psql image the hivesense # Dockerfile's PSQL_CLIENT_VERSION names, so `install` takes install_app.sh's # advisory-lock path (it needs only python3 and psycopg2) - Between the
postgrestandshellcheckFROMlines, add:FROM registry.gitlab.syncad.com/hive/common-ci-configuration/psql:3839feb0c30846aa3ae262143f216a3c1a0fa20b@sha256:6a3794d9fdc4e0590789fc01e48539bcd73273715e3b69dd1f45d3ecde8b1cde AS psql - After
COPY --from=shellcheck …, add:COPY --from=psql /usr/local/bin/install_with_app_lock.py /usr/local/bin/install_with_app_lock.py
- After the
- Embedding server image (the
ollamaanchor in.aidev/test-stack.compose.yml):registry.gitlab.syncad.com/hive/hivesense/aidev-ollama@sha256:933e772d48d20452951670b45903b4ba782deadd55256bcdab98d12458590d88. It serves bothall-minilm:l6-v2andhf.co/mixedbread-ai/mxbai-embed-xsmall-v1:F16(384 dims, MiniLM tokenizer, so the existingall-MiniLM-L6-v2tokenizer.json serves it too). Make these changes to.aidev/runtime/ollama.Dockerfile:- Replace the header line
# copies it there through a volume). Build and pin with .aidev/runtime/build.sh.with:# copies it there through a volume). Also CI's slice-mode model, # hf.co/mixedbread-ai/mxbai-embed-xsmall-v1:F16 (Matryoshka-trained, 384 dims, MiniLM # tokenizer, so the same tokenizer.json serves both). Build and pin with # .aidev/runtime/build.sh. - Replace
ollama pull all-minilm:l6-v2 && ollama list; \with these two lines:ollama pull all-minilm:l6-v2 \ && ollama pull hf.co/mixedbread-ai/mxbai-embed-xsmall-v1:F16 && ollama list; \
- Replace the header line
- Paging and index checks in
embed-search(full mode). Run these against the seeded, embedded corpus in.aidev/run-checks.sh/.aidev/hivesense_smoke.py:find_nearest_posts_with_embeddingwith_limit2 returns 2 rows, in the same order the one-shot variant returns them.- With
_limit100 it returns every embedded seeded post and nothing else. - Continuing with
_start_post_idset to the second result returns the remainder: no repeats and no gaps compared with the one-shot order. - The index named by
get_hnsw_index_name()exists inpg_class.
- The #54 (closed) test, done CI's way. After install, set
hafd.contexts.events_id = 0forhivesense_appand call the realsync_embeddings.ensure_context_detachedfrom the tree with psycopg (the runtime image has it). Then assert thatevents_id = hive.unreachable_event_id(). Mirror CI's failure messages. - Slice mode. Reinstall with
--llm=hf.co/mixedbread-ai/mxbai-embed-xsmall-v1:F16 --vector_size=384 --tokenizer-model=all-MiniLM-L6-v2 --use-reduced-embeddings=true --reduction-mode=slice --reduced-dim=256, plus CI's query prefix (.gitlab-ci.yml:441-443) and the other optionsembed-searchalready uses. Embed the seeded posts and assert:- reduced vectors of the reduced dimension exist for every long seeded post and none for the short one;
- the slice-mode HNSW index (
get_hnsw_index_name()) exists; - the paging checks from 3 pass in slice mode (this is the #56 (closed) case);
- the endpoint rank checks pass. If an assertion that holds for all-minilm doesn't hold for mxbai, weaken that assertion for slice mode only and note why in the README.
- PCA mode. Install with
--reduction-mode=pca, a small--reduced-dim, and--reduced-matrix-sourcepointing at a matrix file the step generates. Assert that reduced vectors of that dimension are stored and that search returns the embedded seeded posts. - Lock wrapper. With the wrapper in the image, a normal
installmust go through it: its log contains nowrapper not foundWARNING. Add a skip-path check modelled on CI'slock_skip_test. Hold the app's shared install lock from a background psql session, then runinstall_app.sh. It must exit 0, print theSkipping hivesense install … lock held byline, and leave the installed schema untouched. Release the lock afterwards, including on failure. - Stack-pin drift step (stackless, in
quick,fullandcanary). It fails when the image pins in.aidev/test-stack.compose.ymldon't correspond to.gitlab-ci.yml'sUPSTREAM_BRANCH, naming the stale image and the expected branch. One way to do this: carry the tag next to each digest (repo:tag@sha256:…; the release tag for hafah, reputation_tracker and hivemind, and the HAF commit tag plus a recorded branch for HAF), then compare those toUPSTREAM_BRANCH. The step must not need network access. - README. Update
.aidev/README.md's step and slot tables, the "Not covered" paragraph and the image sections to match. Add a section saying that GitLab CI remains the gate for what AIDEV can't run: hivesense image builds (build_images), block processing (main_loop.sqlscheduler/worker loop,process_blocks.sh,hivesense_block_processor.py), the 1M-block sync,run_pca.py, the rebase rehearsal (legacy_rebase.sql,hivesense_rebase_test.sh), the PostgREST rewriter and the JMeter benchmarks. Before each merge ofaidev/integrationintodevelop, run a web- or API-triggered pipeline onaidev/integration(theworkflow:rules allow exactly that,.gitlab-ci.yml:57) and merge only when it is green.
Context and decisions
- Eric proposed items 1-4 and the CI gate in 9. Dan accepted all of them on 2026-10-15. The drift step (8) was proposed by Claude and accepted by Dan in the same exchange.
- Building images, block processing and the rebase rehearsal need docker-in-docker or a block log. Dan decided they stay in GitLab CI behind the pre-merge pipeline step rather than moving into AIDEV now. Moving image builds into AIDEV is possible later work and is out of scope here.
- The images were pushed by hand beforehand, as for #61 (closed), so the workflow needs no registry write access. Verified on 2026-10-15:
- the tests image has the wrapper, and
import psycopg2works; - the ollama image lists both models with
--network none; - mxbai returns 384-dim embeddings offline.
- the tests image has the wrapper, and
- Proposed, not confirmed: generate the PCA matrix in the step (for example a deterministic orthonormal projection, or one computed from the full-mode embeddings in plain Python) rather than running
run_pca.py. The runtime image has no numpy/sklearn/matplotlib, and four posts would give a degenerate fit anyway.run_pca.pystays CI-only. The worker may choose a different matrix source, provided the image doesn't need rebuilding. - Proposed, not confirmed: put slice and PCA in their own steps (for example
embed-slice,embed-pca) so their failures are reported separately. Each step ends by reinstalling the default configuration, asembed-searchdoes.
Acceptance criteria
- With the new digests pinned, every step of the
fullslot passes on the candidate. - If
find_nearest_posts_with_embedding's continuation (_start_post_id) skips or repeats a result, or returns more than_limitrows,embed-searchfails, naming the paging call. The same breakage in slice mode fails the slice check. - If the HNSW index
get_hnsw_index_name()names is missing (full or slice mode), the run fails, naming the index. - If
ensure_context_detachedno longer setsevents_idtohive.unreachable_event_id(), the run fails with the #54 (closed) message. - In slice mode, the seeded long posts have reduced vectors of
--reduced-dimdimensions and the short post has none. In PCA mode, the reduced vectors have the configured dimension and search returns the embedded seeded posts. - A normal
installlog contains noinstall_with_app_lock.py wrapper not foundWARNING. While another session holds the app's install lock,install_app.shexits 0, logs the skip line and leaves the installed schema untouched. - If
UPSTREAM_BRANCHin.gitlab-ci.ymlchanges and any of the four stack images isn't re-pinned, thequickslot fails, naming the image. When everything matches, it passes. .aidev/README.mddocuments the new steps and both image changes, lists what stays CI-only, and states that a green web/API pipeline onaidev/integrationis required before merging intodevelop.
Verification
Run the full slot on the candidate. For each new check, also show that it can fail: break the relevant code in a scratch copy and run the step against it. Examples: make the paging function's continuation skip a row, drop the events_id UPDATE in ensure_context_detached, drop the index, edit UPSTREAM_BRANCH, and remove the wrapper path (the step must fail on the WARNING). Record each failing output in the workflow log. Don't commit the breakages.
Integration points
.aidev/runtime/Dockerfile— add the wrapper exactly as above.aidev/runtime/ollama.Dockerfile— pull the slice model exactly as above.aidev/project.yaml— pinenvironment.imageto the new tests digest; slot args for the new steps.aidev/test-stack.compose.yml— pin the new ollama digest; record tags so pin drift can be checked.aidev/run-checks.sh— paging/index, #54 (closed), slice, PCA, lock-skip and pin-drift checks.aidev/hivesense_smoke.py— mode-aware search checks.aidev/README.md— steps, images, CI-only list, pre-merge pipeline rule
Scope boundary
No changes to hivesense's own SQL, scripts, Dockerfile or .gitlab-ci.yml. If a new check finds a real hivesense defect, file a separate issue for it. Don't fix it here, and don't weaken the check to make it pass. Moving image builds or block processing into AIDEV is out of scope.