[hfm] slow hive.set_irreversible
On steem-6 hive.set_irreversible last 900ms. The situation: hived is connected with haf (sql_serializer) and works in live sync mode ( there were more than 57m of blocks ) hafah is synchronizing with haf -> its contexts exists and is far before the head block, hafah is working on detached mode
Because possible race conditions between (removing obsolete events and reversible data by hived) and (taking next event and reversible data by the applications), events and reversible data for blocks greater than the current block of the slowest contexts are not removed.
Because the hafah's current block is low, then reversible data and obsolete events generated during live sync are not removed and there were stay in the db ( for example there were millions of operations in hive.operations_reversible table ).
Function hive.set_irreversible is responsible to move reversible data to irreversible tables and remove events because it is a place where the haf is notified about considering new blocks as irreversible. A large number of reversible data is suspected to be a root cause of slowin set_irreversible