deadlock on fork_database mutex
This problem appears after recent changes in locking, first failing is hived build on top of 3efd09a3 (although is hard for me to say where exactly is the problem). When fork happens, there is chance of deadlock. It doesn't happen 100% times, that's why pipelines after commit 3efd09a3 passed (it's good idea to add tests failing at 100% rate when this happens). However it is impossible to upgrade haf with newest version of hive as submodule, CI fails there almost always.
Last message from default logger is:
2022-04-14T15:08:58.973107 database.cpp:1158 _push_block ] Switching to fork: 000000750743259f936e4272d8de30728c69e2ce
Last message from chainlock logger:
2022-04-14T15:08:58.973 Δ 10µs message read_loop with_read_lock ] trying to get fork_read_lock, read_lock_count=0 write_lock_count=0 thread id=5796905013781373106 fork_database.hpp:151
2022-04-14T15:08:58.973 Δ 3µs message read_loop with_read_lock ] got fork_read_lock: read_lock_count=1 write_lock_count=0 fork_database.hpp:166
2022-04-14T15:08:58.973 Δ 5µs ?unnamed? with_write_lock ] trying to get fork_write_lock: read_lock_count=0 write_lock_count=0 thread id=18423280004977553937 fork_database.hpp:176
2022-04-14T15:08:58.973 Δ 16µs message read_loop with_read_lock ] trying to get fork_read_lock, read_lock_count=1 write_lock_count=1 thread id=5796905013781373106 fork_database.hpp:151
This job https://gitlab.syncad.com/hive/haf/-/jobs/283810 is an example of failure. HAF system tests are being run with faketime but it is not related to this issue (without faketime we also have this problem, reproduced on my machine)