Undo changes from failing custom operation execution even when not `is_producing`.
See how delegate_rc_operation
is handled to find the places in code that needs to be addressed.
Currently when custom json operation is handled inside hived
plugin (like above mentioned operation in rc_plugin
), it can throw exception like any other evaluator and it will cause transaction being processed to fail. As a result all the state changes are undone and both hived
as well as plugin return to clean consistent state. Except... it only happens when the node is_producing
- see void custom_json_evaluator::do_apply( const custom_json_operation& o )
. When node is taking transactions from blocks made by other witnesses or simply reapplies pending transactions, exception from custom operation will lead to plugin state being inconsistent.
Example: delegate_rc_operation
first creates new or modifies existing RC delegations for all targeted delegatees collecting delegation delta in the process, only then it checks if the delta can actually be applied to delegator "RC balance". If delta is too big, there is an exception. It means that it only takes one rogue witness (or not even rogue, it might just have slightly different RC balance for delegator account for whatever reason - it is nonconsensus after all) to put rc_plugin
in inconsistent state, where delegations are present, but the source of those delegations is untouched. The inconsistency will spread into the future, since now that broken node will itself accept transactions that use "illegal" delegations, it will touch delegator account when delegations are removed and so on. The broken node will therefore "go rogue", becoming potential source of problems for other nodes. The above situation can even happen completely naturally as a result of different transaction order inside pending list vs actual block (see rc_plugin_tests/rc_tx_order_bug
test scenario), although in that case the resulting inconsistency will only persist until problematic transaction is removed from pending list.
The solution is to wrap custom json evaluation in its own undo session that would be squashed
when all works ok. See database::_push_transaction
. Note that even if there is no session at the start (f.e. during replay), we still want to do it (yes, it will make replay with many such operations slower), otherwise we would be protected during live sync only to become broken during replay. The undo session is there to return plugin to consistent clean state, as if failing operation never happened.