Remove replay_data_copy job and simplify sync
Summary
- Remove redundant
replay_data_copyjob entirely (saves ~5 min per pipeline) - Refactor
syncjob to handle HAF cache fetching directly - Use
copy_datadir.shfrom HAF submodule (same pattern as balance_tracker) - Remove broken
cache-manager copylogic that was causing failures
Changes
-
Deleted
replay_data_copyjob - was duplicating work that sync already does -
Simplified
syncbefore_script:- Initialize HAF submodule (recursive for nested hive submodule)
- Fetch HAF cache from NFS if needed via cache-manager
- Use
copy_datadir.shfor data copy (handles permissions properly)
-
Updated dependent jobs:
-
cleanup_pipeline_cache: removedreplay_data_copyfrom needs -
e2e_benchmark_on_postgrest: removedreplay_data_copyfrom needs, removed unusedHIVED_UID
-
Why
The previous flow had:
-
prepare_haf_data→ creates HAF cache -
replay_data_copy→ copies HAF cache to pipeline-specific location -
sync→ tries to copy AGAIN using brokencache-manager copy
Now (like balance_tracker):
-
prepare_haf_data→ creates HAF cache -
sync→ fetches from NFS if needed, usescopy_datadir.sh, runs sync
Testing
Pipeline will verify the refactored flow works correctly.