Implemented concurrent pipelines on Gitlab CI
Implemented concurrent pipelines on Gitlab CI this way:
- We have 2 gitlab-runner instances on 2 physical hosts.
- gitlab-runner instances run with docker executors and run jobs concurrently up to defined limits. For tag
hivemind-light-job
limit is about 6, and for taghivemind-heavy-job
limit is about 4. Limits can be set in file/etc/gitlab-runner/config.toml
on runner's physical host. - Runners can be pointed to databases and hived servers via variables on CI/CD settings page.
- Databases are created for each pipeline. They will be removed by a cron task after some predefined time (not implemented yet).
- Each test gets his own instance of hivemind server, so we don't need stages starting and stopping hivemind server.
- Stage
build
used to build hivemind python egg was removed – it's unnecessary.
Also:
- Added command line parameters enabling showing timestamps or epochs in log statements. It's useful when somebody wants to pass log file to an analysis tool.
- Added command line parameter enabling masking sensitive data in log statements. It's useful when log can be seen by untrusted audience, like on CI/CD server.
Warning
Only one of two newly created runners is enabled now. I have to resolve problem with stages in gitlab-ci.yaml
. Concurrent runners on different physical host pick jobs, and it is possible, that runner doesn't know which database server should be used. We have two possibilities here:
- Run all jobs in one stage (hivemind-sync + e2e-tests). Caveat: difficult reporting, difficult handling of failures.
- Have multiple stages, but pass info about database server via artifacts. Caveat: some tests will be pointed to the database server running on the other physical machine (slowness).
Edited by Wojciech Barcik