Workaround TestTools' stability problems

Piotr Batko requested to merge pbatko/fix-unstable-ci into master

When hived is heavily loaded with work, it can sometimes:

  1. ignore some transactions and never add them to blocks,
  2. fail to access database to perform requested operation and respond with "Unable to acquire database lock" error.

I discovered these problems in two situations. First case is on CI, when I run multiple pipelines at the same time. Second case is on my local computer, when I run tests with multiple (200k) accounts creation.

Proposed workarounds will retry to send request forever if above errors occur. Because finally transaction will be appended to block (1) and finally node will acquire database lock (2). These workarounds could be reverted when problem will be mitigated in hived.

I rewrite one test (which is kind of performance test), which creates multiple (200k) accounts to always cause one of above error. Here I run it multiple times to make sure it always fails (pipeline #42855, pipeline #42857, pipeline #42858, pipeline #42859, pipeline #42860). Then I apply workarounds and it helped, but one of five pipelines froze probably because of hive#263 (closed). I applied fixes from hive@252cf962 (this commit is already merged with origin/develop) and all problems have disappeared. All run pipelines passed (pipeline #42932, pipeline #42934, pipeline #42935, pipeline #42936, pipeline #42937).

Edited by Piotr Batko

Merge request reports