Draft: CI speedup and stability fixes
Summary
This MR contains CI/CD improvements focused on speed optimizations and stability fixes.
Changes
CI Stability Fixes
- Fix validate_response file concatenation issue in REST API tests
- Remove erroneous 'auto' argument from smoketest calls that was causing failures
- Make benchmark-results-collector trigger manual to avoid blocking pipelines
- Add hive-builder-9 tag to cleanup jobs for consistency
CI Speed Optimizations
- Increase git submodule parallelism from 4 to 8 jobs
- Increase pytest workers from 4 → 8 → 16 → 12 (settled on 12 for optimal balance)
- Add explicit RUNNER_PYTEST_WORKERS variable for parallel test configuration
- Reduce initial sleep from 20s to 5s in sync scripts
- Add pip/tox caching for e2e_benchmark_on_postgrest job
PostgreSQL Tuning for CI
- WAL tuning: checkpoint_completion_target optimization
- Parallel workers: max_parallel_maintenance_workers, max_parallel_workers
- SSD optimizations: random_page_cost, effective_io_concurrency
- Tested and removed settings that didn't help (jit=off, commit batching with fsync=off)
Test Results
Commits
19 commits total, iteratively tested and refined through multiple pipeline runs.
Edited by Dan Notestein