Re-enable API benchmark script in CI

Dan Notesteinrequested to merge
fix/reenable-benchmarks into develop

Summary

Re-enable the API benchmark script that was disabled in September 2024 (MR !717 (merged)).

Status: Pipeline still failing with script_failure despite all tests passing. Needs further investigation.

Changes Made

  1. c1e9fea2 - Re-enabled the api-benchmark-script that was commented out during PostgREST migration
  2. 62dd9535 - Added conditional check to skip csv-report-parser if benchmark.csv doesn't exist (PostgREST doesn't support server-side timing)
  3. 7dae2b1a - Fixed $HIVEMIND_BENCHMARKS_IDS_FILE variable reference, reduced iterations from 5 to 1

Investigation Findings

  • All benchmark tests pass (JUnit shows failures="0")
  • benchmark.csv is never created because PostgREST doesn't support request timing
  • Script still fails despite tests passing

Recommendations

To unblock:

  1. Remove set -e temporarily to identify failing command
  2. Check job trace in GitLab web UI for actual error

Long-term:

  1. Update tavern-time-request library to record full round-trip timing
  2. Consider if benchmarks are still valuable without server-side timing
  3. Alternative: use PostgreSQL log_min_duration_statement for timing

Files Modified

  • scripts/ci/start-api-benchmarks.sh
Edited by Dan Notestein

Merge request reports