Make RC plugin less noisy

RC plugin currently emits tons of spammy messages during the replay, such as:

Accepting transaction by alice, has 1 RC, needs 2 RC, block 3, witness bob.

Quick solution is to make it go to the debug log.

Another thing is do we really need to check on RC during replay? That's plenty of checks that were done before, and non consensus anyway.

added Doing question suggestion labels

assigned to @gandalf

FYI: the main reason we're seeing these messages and a potential (but painful) fix is described here: #197 (closed) In summary, it's mostly because some bots are generating 2 or more transactions per block when they have nearly exhausted their RC to the point where all their transactions can't be paid for with remaining RC, and the transactions arrive in different order on different nodes. It should be noted that in this case the message is actually erroneous: the transaction it complains about was not accepted by the producing witness and has not become part of any block.

This check is performed after every on_post_apply_transaction. This is the code block that generates this warning log message: https://gitlab.syncad.com/hive/hive/-/blob/master/libraries/plugins/rc/rc_plugin.cpp#L272

The problem with moving these log messages to debug level is that when they are not erroneous, they are useful to see, because they can alert to real issues. Maybe we should create a simple heuristic to filter out most of the erroneous ones like this:

bool almost_has_mana(int64_t mana_needed)
  return current_mana * 2 >= mana_needed;

if !has_mana
  if almost_has_mana(mana_needed)
    dlog
  else
    ilog

We could test it on a replay to see if it does a reasonable job of eliminating them. If it doesn't filter enough, we could increase multiplication factor to 3 or more.

The noise has been caused by bug introduced in RC delegations related changes. It is fixed already: !346 (merged)

closed

@bwrona I'm reopening this for discussion because !346 (merged) doesn't seem to help, maybe combining suggested simple heuristic with dlog if it's during replay / resync (i.e. far from head block), ilog otherwise could improve that?
thing is that after a replay of consensus node I have 538409 lines of log, where 537459 are matching:
rc_plugin.cpp:271 operator() ] Accepting transaction by

(that while running current develop)

reopened

removed Doing label

added To Do label

assigned to @ABW and unassigned @gandalf

As gandalf indicated above, I believe the heuristic filter is still necessary.

There were three sources of "Accepting..." warnings.

• First is from historical operations that were accepted during turbulent times of constant changes in RC in first weeks of RC being introduced. Almost all messages of that type fit into this category. Note that after we change the RC costs there will be even more messages (like 200 times more), therefore I decided to turn them off completely until after HF26; but before that happens two other sources described below were fixed
• Second is from incorrect interpretation of reality by the affected node: it would generate such warnings based on its own application of pending transactions, but would attribute it to witness that recently produced block; it was fixed in !348 (merged)
• Third is from block producer including transactions in blocks despite their lack of RC (which means the warnings were actually properly showing a bug); fixed in !476 (merged); it was caused by combination of the fact that is_producing()==false when producing block and mechanism similar to above second source - when transaction became pending it would not be dropped with a mere lack of RC