API tests execution time reports

Added stage with benchmark tests for hivemind api calls.

• benchmark_generator.py will create set of benchmark tests basing on tavern.yaml tests in tests_api directory.
• Benchmarks will create json file which is parsed by json_report_parser.py. The html report is generated.
• Report contains data such as test name and mean execution time of a test (5 runs of each call).
• If mean time is greater than threshold the job running that test will fail.

added 13 commits

• d4776f8f...2bf91652 - 12 commits from branch develop
• 1f84c88e - Rebase to current develop

Compare with previous version

added 1 commit

• 822fb5c6 - Reports will include call parameters for api calls that are above treshold

Compare with previous version

added 1 commit

• 69b0b85f - Better class to file name conversion

Compare with previous version

changed the description

added 1 commit

• 18808d94 - Time threshold can be configured via command line parameter

Compare with previous version

unmarked as a Work In Progress

added 86 commits

• 18808d94...7f0fb4d6 - 85 commits from branch develop
• 3c8a7eb3 - Rebase to current develop

Compare with previous version

added 1 commit

• d177b8a3 - Added benchmarks

Compare with previous version

added 1 commit

• 73718c87 - gitlab-ci fix

Compare with previous version

added 1 commit

• dc8c3f6f - Fixed paths to benchmark generator script

Compare with previous version

added 1 commit

• aabadde8 - tests_api updated

Compare with previous version

added 1 commit

• f78e02f4 - Install missing pytest-benchark

Compare with previous version

added 1 commit

• 5f6d7723 - Benchmarks moved to tox

Compare with previous version

added 1 commit

• 0225d2f5 - Added missing schema

Compare with previous version

added 1 commit

• 5318f66a - Added missing schema

Compare with previous version

added 22 commits

• 5318f66a...47a50445 - 21 commits from branch develop
• 7593701a - Rebase to current develop

Compare with previous version

added 1 commit

• 3cae0aa7 - Fixes in tests_api

Compare with previous version

added 1 commit

• 941487b2 - Script for running benchmarks

Compare with previous version

General question - what about concurrent execution of tavern tests ? Does it is somehow disabled on benchmark spawn ?

resolved all threads

added 1 commit

• 57ebbd35 - Do not allow benchmark tests to fail, failed test is a failed job. Changed...

Compare with previous version

resolved all threads

marked as a Work In Progress

@bwrona please do not merge yet.

added 1 commit

• 632361f0 - Html reports are based on benchmark data not on xml junit data

Compare with previous version

added 2 commits

• 665db8ee - Format test time
• f9e11c47 - Remove debug print

Compare with previous version

unmarked as a Work In Progress

changed the description

@bwrona added some fixes, also reworked html report. Details are in MR description above.

added 14 commits

• f9e11c47...417cd793 - 13 commits from branch develop
• 1e166a3a - Rebase to current develop

Compare with previous version

added 1 commit

• 9e483b83 - Fixed file names of artifacts

Compare with previous version

added 27 commits

• 9e483b83...8d29ca01 - 26 commits from branch develop
• b7fb674b - Rebase to current develop

Compare with previous version

added 1 commit

• 80a18be4 - Switched to proper test branch

Compare with previous version

added 27 commits

• 80a18be4...62e741f3 - 26 commits from branch develop
• b90ae0da - Rebase to current develop

Compare with previous version

added 1 commit

• 5ce6f29f - Add info about failing benchmarks to the console

Compare with previous version

added 1 commit

• f74a6129 - Add comments to json_report_parser.py

Compare with previous version

added 23 commits

• f74a6129...c369519e - 22 commits from branch develop
• 3ff7f47c - Rebase to current develop

Compare with previous version

added 1 commit

• aeb8936e - All benchmarks moved to one job to avoid starting and stoppng server several times

Compare with previous version

added 1 commit

• e4ba9138 - Fixes

Compare with previous version

marked as a Work In Progress

added 1 commit

• 933bf13f - Tests moved to one job to limit starting/stopping hivemind server

Compare with previous version

added 1 commit

• 1568214c - Fix script path, add test env to tox invocation

Compare with previous version

added 1 commit

• fe1d7c4d - Fixed path

Compare with previous version

added 1 commit

• 172d9b2d - Benchmarks script moved to CI dir

Compare with previous version

added 1 commit

• f3885f06 - Added missing pyyaml module

Compare with previous version

added 1 commit

• b9589ff9 - python3 -m pip install instead pip install

Compare with previous version

added 1 commit

• 41571bcd - Benchmarks boilerplate moved to tox

Compare with previous version

added 1 commit

• 1fb1508a - Add missing param

Compare with previous version