Skip to content

Implemented concurrent pipelines on Gitlab CI

Wojciech Barcik requested to merge wb-ci-concurrent-pipelines into develop

Implemented concurrent pipelines on Gitlab CI this way:

  1. We have 2 gitlab-runner instances on 2 physical hosts.
  2. gitlab-runner instances run with docker executors and run jobs concurrently up to defined limits. For tag hivemind-light-job limit is about 6, and for tag hivemind-heavy-job limit is about 4. Limits can be set in file /etc/gitlab-runner/config.toml on runner's physical host.
  3. Runners can be pointed to databases and hived servers via variables on CI/CD settings page.
  4. Databases are created for each pipeline. They will be removed by a cron task after some predefined time (not implemented yet).
  5. Each test gets his own instance of hivemind server, so we don't need stages starting and stopping hivemind server.
  6. Stage build used to build hivemind python egg was removed – it's unnecessary.

Also:

  1. Added command line parameters enabling showing timestamps or epochs in log statements. It's useful when somebody wants to pass log file to an analysis tool.
  2. Added command line parameter enabling masking sensitive data in log statements. It's useful when log can be seen by untrusted audience, like on CI/CD server.

Warning

Only one of two newly created runners is enabled now. I have to resolve problem with stages in gitlab-ci.yaml. Concurrent runners on different physical host pick jobs, and it is possible, that runner doesn't know which database server should be used. We have two possibilities here:

  1. Run all jobs in one stage (hivemind-sync + e2e-tests). Caveat: difficult reporting, difficult handling of failures.
  2. Have multiple stages, but pass info about database server via artifacts. Caveat: some tests will be pointed to the database server running on the other physical machine (slowness).
Edited by Wojciech Barcik

Merge request reports