During sync, massive-sync conditions (dropped indexes, FKs, tables switched to...

Bartek Wronarequested to merge
index_rebuilt_limit into develop

Summary

This MR adds NFS-based caching support for CI pipelines, enabling sharing of HAF replay data and sync data across multiple CI builders. This eliminates the need for each builder to maintain its own local cache and allows tests to run on any available builder.

Changes

NFS Cache Infrastructure

  • Added cache-manager.sh script for managing NFS cache operations (get/put with locking)
  • Added NFS fallback paths in test job entrypoints for cross-builder data access
  • Added tar archive extraction support for NFS cache in test services
  • Use sudo to restore PostgreSQL permissions after NFS cache extraction

CI Pipeline Fixes

  • Fixed pgdata ownership using explicit UID 105:109 instead of username-based resolution
  • Added sudo for copying files to datadir in sync after_script
  • Updated test jobs to check NFS cache when local cache is not available
  • Increased wait timeouts (10m -> 55m) for block processing

HAF Submodule Updates

  • Updated HAF to 465036e7 (fix pgdata ownership check)
  • Updated HAF to 642330a5 (tablespace permissions fix)
  • Updated HAF to e842bc87 (NFS-safe locking)
  • Updated HAF to df653a09 (pgdata permissions fix)
  • Multiple updates for NFS cache manager and BusyBox flock fixes

Patterns Applied

  • Use explicit UIDs (105:109) instead of usernames for postgres ownership to avoid UID mismatch between host and container
  • Use --no-preserve=mode,ownership when copying from NFS to avoid permission issues
  • Replace hardcoded hive-builder-9 with flexible tags

Test Results

Pipeline passing with all tests green.

Edited by Dan Notestein

Merge request reports