From e81aee57043db7017b85bcb70b00556835a358f5 Mon Sep 17 00:00:00 2001 From: Dan Notestein Date: Sun, 4 Jan 2026 00:58:30 -0500 Subject: [PATCH] docs: add Phase 4 migration examples and update guides - Add real migration examples from reputation_tracker (MR !143) and balance_tracker (MR !261) - Update haf-app-testing-templates.md with Phase 4 DinD templates - Add step-by-step migration guide with before/after examples - Update migrated applications table with line reduction stats - Add best practices for project-specific templates --- docs/haf-app-testing-templates.md | 159 +++++++++++++++++++++++------- docs/haf-app-testing.md | 82 +++++++++++++++ 2 files changed, 206 insertions(+), 35 deletions(-) diff --git a/docs/haf-app-testing-templates.md b/docs/haf-app-testing-templates.md index 21bb733..91b2dd1 100644 --- a/docs/haf-app-testing-templates.md +++ b/docs/haf-app-testing-templates.md @@ -6,6 +6,21 @@ Reusable CI templates for HAF-dependent applications (balance_tracker, reputatio The `templates/haf_app_testing.gitlab-ci.yml` provides composable building blocks for common CI patterns across HAF applications. Apps can use `!reference` tags to include only the pieces they need while keeping app-specific logic inline. +## Quick Start + +```yaml +include: + - project: 'hive/common-ci-configuration' + ref: develop + file: '/templates/haf_app_testing.gitlab-ci.yml' + +variables: + # Required aliases for templates + APP_SYNC_CACHE_TYPE: "${MY_APP_SYNC_CACHE_TYPE}" + APP_CACHE_KEY: "${HAF_COMMIT}_${CI_COMMIT_SHORT_SHA}" + HAF_APP_SCHEMA: "myapp" +``` + ## Available Templates ### Change Detection @@ -24,6 +39,12 @@ The `templates/haf_app_testing.gitlab-ci.yml` provides composable building block - `.haf_app_sync_cleanup` - after_script cleanup - `.haf_app_sync_artifacts` - Standard artifacts configuration +### DinD Test Templates (Phase 4) +- `.haf_app_dind_test_base` - Base DinD test template with cache extraction and compose +- `.haf_app_extract_test_cache` - Extract sync cache for test jobs +- `.haf_app_wait_for_postgres` - Wait for PostgreSQL with timeout +- `.haf_app_wait_for_postgrest` - Wait for PostgREST with DNS fix + ### Tavern Test Components - `.tavern_test_variables` - pytest workers, tavern version, directories - `.tavern_install_deps` - Install tavern in a venv @@ -35,14 +56,87 @@ The `templates/haf_app_testing.gitlab-ci.yml` provides composable building block - `.skip_on_auto_skip` - Skip job when AUTO_SKIP_SYNC=true - `.skip_on_cached_data` - Skip on either condition -## Usage Example +## Migration Examples + +### reputation_tracker (MR !143) +**Before:** 168 lines of inline `.test-with-docker-compose` ```yaml -include: - - project: 'hive/common-ci-configuration' - ref: develop - file: '/templates/haf_app_testing.gitlab-ci.yml' +.test-with-docker-compose: + extends: .docker_image_builder_job_template + stage: test + image: registry.gitlab.syncad.com/hive/reputation_tracker/ci-runner:docker-24.0.1-8 + variables: + HAF_DATA_DIRECTORY: ${CI_PROJECT_DIR}/${CI_JOB_ID}/datadir + HAF_SHM_DIRECTORY: ${CI_PROJECT_DIR}/${CI_JOB_ID}/shm_dir + # ... many more lines + before_script: + # 60+ lines of cache extraction, compose startup, service wait + after_script: + # 30+ lines of cleanup +``` +**After:** 8 lines extending templates +```yaml +.test-with-docker-compose: + extends: + - .haf_app_dind_complete_test + image: registry.gitlab.syncad.com/hive/reputation_tracker/ci-runner:docker-24.0.1-8 + variables: + COMPOSE_OPTIONS_STRING: "--file docker-compose-test.yml --ansi never" +``` + +### balance_tracker (MR !261) + +balance_tracker created a project-specific `.btracker-dind-test` template that consolidates its DinD patterns, then simplified three test jobs. + +**Before:** ~445 lines across 3 jobs +```yaml +regression-test: + extends: .docker_image_builder_job_template + stage: test + image: registry.gitlab.syncad.com/hive/balance_tracker/ci-runner:docker-24.0.1-17 + variables: + HAF_DATA_DIRECTORY: ${CI_PROJECT_DIR}/${CI_JOB_ID}/datadir + # ... many variables + before_script: + # 80+ lines: docker login, cache extraction, blockchain handling, + # ci.env creation, compose startup, PostgreSQL wait + script: + - ./accounts_dump_test.sh --host=docker + after_script: + # 20+ lines of cleanup +``` + +**After:** ~91 lines total +```yaml +# Project-specific base template (consolidates common patterns) +.btracker-dind-test: + extends: .docker_image_builder_job_template + stage: test + image: registry.gitlab.syncad.com/hive/balance_tracker/ci-runner:docker-24.0.1-17 + variables: + BTRACKER_TEST_CACHE_TYPE: "${BTRACKER_SYNC_CACHE_TYPE}" + BTRACKER_TEST_CACHE_KEY: "${BTRACKER_CACHE_KEY}" + WAIT_FOR_POSTGREST: "false" + # ... before_script, after_script consolidated here + +# Simplified job +regression-test: + extends: .btracker-dind-test + needs: + - detect_changes + - sync + - docker-setup-docker-image-build + - prepare_haf_image + script: + - cd "${CI_PROJECT_DIR}/tests/account_balances" + ./accounts_dump_test.sh --host=docker +``` + +## Sync Job Example + +```yaml sync: extends: - .docker_image_builder_job_template @@ -63,36 +157,15 @@ sync: ## Migrated Applications -| Application | Branch | Status | -|-------------|--------|--------| -| reputation_tracker | develop | Merged | -| balance_tracker | develop | Merged | -| hafah | - | Not started | -| hivemind | - | Not started | -| hivesense | - | Not started | -| nft_tracker | - | Not started | -| haf_block_explorer | - | Not started | - -## Future Plans - -### Short-term -1. **Template `.test-with-docker-compose`** - The test job pattern from reputation_tracker that extracts cache, starts compose, waits for services. Could be reused across apps. - -2. **Migrate hafah** - hafah has similar patterns but runs pytest inside a docker container rather than in the CI job. May need variant templates. - -3. **Migrate hivemind** - More complex app with multiple test types, but could use sync templates. - -### Medium-term -4. **Template docker build jobs** - The `.docker-build-template` pattern is similar across apps. - -5. **Template prepare_haf_image/prepare_haf_data** - These jobs are nearly identical across apps. - -6. **Add cache cleanup templates** - Automated cleanup of old sync caches. - -### Long-term -7. **Unified HAF app CI base** - A single extensible template that apps configure rather than compose from pieces. - -8. **Cache warming jobs** - Proactive cache population for common HAF commits. +| Application | Status | MR | Line Reduction | +|-------------|--------|-----|----------------| +| reputation_tracker | ✅ Done | !143 | 168 → 8 lines | +| balance_tracker | ✅ Done | !261 | 445 → 91 lines (net -216) | +| hafah | Pending | - | - | +| hivemind | Pending | - | - | +| hivesense | Pending | - | - | +| nft_tracker | Pending | - | - | +| haf_block_explorer | Pending | - | - | ## Variable Reference @@ -108,3 +181,19 @@ sync: ### For QUICK_TEST mode - `QUICK_TEST` - Set to "true" to enable - `QUICK_TEST_HAF_COMMIT` - HAF commit to use for cache + +### For DinD test templates +- `HAF_DATA_DIRECTORY` - Where to extract cache data +- `HAF_SHM_DIRECTORY` - Shared memory directory +- `COMPOSE_OPTIONS_STRING` - Docker compose options +- `WAIT_FOR_POSTGREST` - Set to "true" to wait for PostgREST + +## Best Practices + +1. **Use consistent cache type prefixes**: Prefix with `haf_` for automatic pgdata permission handling + +2. **Create project-specific templates**: If your app has multiple similar DinD jobs, create a local template (like `.btracker-dind-test`) that extends the common templates + +3. **Use `!reference` for composability**: Build before_script from template components + +4. **Add variable aliases**: Define `APP_SYNC_CACHE_TYPE`, `APP_CACHE_KEY`, `HAF_APP_SCHEMA` at the global level for template compatibility diff --git a/docs/haf-app-testing.md b/docs/haf-app-testing.md index ac601c1..18eb513 100644 --- a/docs/haf-app-testing.md +++ b/docs/haf-app-testing.md @@ -443,6 +443,88 @@ This allows: ## Migration Guide +### Step 1: Add Variable Aliases + +Add these global variables for template compatibility: + +```yaml +variables: + # Aliases for common-ci templates + APP_SYNC_CACHE_TYPE: "${MY_APP_SYNC_CACHE_TYPE}" + APP_CACHE_KEY: "${HAF_COMMIT}_${CI_COMMIT_SHORT_SHA}" + HAF_APP_SCHEMA: "myapp" +``` + +### Step 2: Migrate DinD Test Jobs + +**Before (reputation_tracker, 168 lines):** +```yaml +.test-with-docker-compose: + extends: .docker_image_builder_job_template + before_script: + - !reference [.docker_image_builder_job_template, before_script] + - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY + - | + # 60+ lines of cache extraction + CACHE_MANAGER="/tmp/cache-manager.sh" + curl -fsSL "..." -o "$CACHE_MANAGER" + chmod +x "$CACHE_MANAGER" + "$CACHE_MANAGER" get "${CACHE_TYPE}" "${CACHE_KEY}" "${JOB_DIR}" + # ... blockchain handling, compose startup, service wait + after_script: + - # 30+ lines of cleanup +``` + +**After (8 lines):** +```yaml +.test-with-docker-compose: + extends: .haf_app_dind_complete_test + image: registry.gitlab.syncad.com/hive/reputation_tracker/ci-runner:docker-24.0.1-8 + variables: + COMPOSE_OPTIONS_STRING: "--file docker-compose-test.yml --ansi never" +``` + +### Step 3: For Apps with Multiple DinD Jobs + +If your app has several DinD test jobs (like balance_tracker), create a project-specific template: + +```yaml +# Project-specific template consolidating common patterns +.myapp-dind-test: + extends: .docker_image_builder_job_template + stage: test + image: my-ci-runner-image + variables: + HAF_DATA_DIRECTORY: ${CI_PROJECT_DIR}/${CI_JOB_ID}/datadir + MYAPP_TEST_CACHE_TYPE: "${MYAPP_SYNC_CACHE_TYPE}" + before_script: + - !reference [.docker_image_builder_job_template, before_script] + - !reference [.haf_app_extract_test_cache, script] + # ... compose startup, service wait + after_script: + # ... cleanup + +# Simplified jobs +regression-test: + extends: .myapp-dind-test + script: + - ./run-tests.sh + +pattern-test: + extends: .myapp-dind-test + variables: + WAIT_FOR_POSTGREST: "true" + script: + - pytest tests/ +``` + +### Real Migration Examples + +| Project | MR | Before | After | Reduction | +|---------|-----|--------|-------|-----------| +| reputation_tracker | !143 | 168 lines | 8 lines | -160 lines | +| balance_tracker | !261 | 445 lines (3 jobs) | 91 lines | -354 lines | + ### From inline cache-manager fetch **Before:** -- GitLab