HAF Crash on fork
Only one of the nodes in the cluster crashed, no issues on other nodes. Same setup.
2023-02-14T16:07:00.368612 full_transaction.cpp:506 boost_se_body ] full_transaction_cache lock contention, waited 24µs, lock has been in use 610034 of 300047948 times it was used
2023-02-14T16:07:01.391380 data_processor.cpp:153 only_report_batch_fi ] 72305696 commited by Transaction multisig data writer
2023-02-14T16:07:01.391396 data_processor.cpp:153 only_report_batch_fi ] 72305696 commited by Accounts data writer
2023-02-14T16:07:01.404502 block_num_rendezvous_trigger.cpp:40 report_complete_thre ] Dump whole block 72305696
2023-02-14T16:07:01.405512 p2p_plugin.cpp:188 handle_block ] Got 265 transactions on block 72305696 by ura-soul -- Block Time Offset: 1405 ms
2023-02-14T16:07:01.481850 p2p_plugin.cpp:580 broadcast_transactio ] Broadcasting tx #a540243620bfb3c87c7731acbc3b263624fb7be8
2023-02-14T16:07:02.851385 database.cpp:1070 _maybe_warn_multiple ] Encountered block num collision at block 72305696 due to a fork, witnesses are: [["ura-soul","2023-02-14T16:07:00"],["blocktrades","2023-02-14T16:07:03"]]
2023-02-14T16:07:02.851424 p2p_plugin.cpp:188 handle_block ] Got 266 transactions on block 72305696 by blocktrades -- Block Time Offset: -148 ms
2023-02-14T16:07:04.252059 p2p_plugin.cpp:580 broadcast_transactio ] Broadcasting tx #feb885fa989504af48d607a3a4eda563e919641a
2023-02-14T16:07:05.677389 database.cpp:1230 _push_block ] calling switch_forks() from _push_block()
2023-02-14T16:07:05.677410 database.cpp:1079 switch_forks ] Switching to fork: 044f4c21498dfaa310f5bb2cf17506527af7f9c8
2023-02-14T16:07:05.677415 database.cpp:1082 switch_forks ] Before switching, head_block_id is 044f4c201f80699e2b995fde7178eac6f4bde24d head_block_number 72305696
2023-02-14T16:07:05.677423 database.cpp:1085 switch_forks ] Destination branch block ids:
2023-02-14T16:07:05.677427 database.cpp:1087 operator() ] - 044f4c21498dfaa310f5bb2cf17506527af7f9c8
2023-02-14T16:07:05.677431 database.cpp:1087 operator() ] - 044f4c2052651a25ba746e5c6370db3068c9b901
2023-02-14T16:07:05.677435 database.cpp:1092 switch_forks ] - common_block_id 044f4c1fc9712ad7c572c4bea68da35853af07e1 common_block_number 72305695 (block before first block in branch, should be common)
2023-02-14T16:07:05.677439 database.cpp:1096 switch_forks ] Source branch block ids:
2023-02-14T16:07:05.677442 database.cpp:1098 operator() ] - 044f4c201f80699e2b995fde7178eac6f4bde24d
2023-02-14T16:07:05.677446 database.cpp:1107 switch_forks ] - Popping block 044f4c201f80699e2b995fde7178eac6f4bde24d
2023-02-14T16:07:05.679828 database.cpp:1109 switch_forks ] - Popped block 044f4c201f80699e2b995fde7178eac6f4bde24d
2023-02-14T16:07:05.679837 database.cpp:1111 switch_forks ] Done popping blocks
2023-02-14T16:07:05.682410 database.cpp:1121 switch_forks ] pushing block #72305696 from new fork, id 044f4c2052651a25ba746e5c6370db3068c9b901
2023-02-14T16:07:05.687275 data_processor.cpp:153 only_report_batch_fi ] 72305696 commited by Transaction multisig data writer
2023-02-14T16:07:05.687286 data_processor.cpp:153 only_report_batch_fi ] 72305696 commited by Accounts data writer
2023-02-14T16:07:05.699035 block_num_rendezvous_trigger.cpp:40 report_complete_thre ] Dump whole block 72305696
2023-02-14T16:07:05.699885 database.cpp:1121 switch_forks ] pushing block #72305697 from new fork, id 044f4c21498dfaa310f5bb2cf17506527af7f9c8
2023-02-14T16:07:05.714043 data_processor.cpp:153 only_report_batch_fi ] 72305697 commited by Transaction multisig data writer
2023-02-14T16:07:05.714055 data_processor.cpp:153 only_report_batch_fi ] 72305697 commited by Accounts data writer
2023-02-14T16:07:05.738082 block_num_rendezvous_trigger.cpp:40 report_complete_thre ] Dump whole block 72305697
2023-02-14T16:07:05.739428 database.cpp:1191 switch_forks ] done pushing blocks from new fork
2023-02-14T16:07:05.739448 database.cpp:1078 boost_se_body ] Done fork switch
2023-02-14T16:07:05.739486 p2p_plugin.cpp:188 handle_block ] Got 326 transactions on block 72305697 by guiltyparties -- Block Time Offset: -260 ms
2023-02-14T16:07:05.831793 data_processor.cpp:212 handle_exception ] Data processor hive.set_irreversible caller detected SQL statement execution failure. Failing statement: `SELECT hive.set_irreversible(72305696)'.
2023-02-14T16:07:05.831817 data_processor.cpp:16 kill_node ] An error occured and HAF is stopping synchronization...
2023-02-14T16:07:05.832532 application.cpp:93 handle_signal ] _last_signal_code: 2
2023-02-14T16:07:05.832815 application.cpp:84 generate_interrupt_r ] interrupt requested!
2023-02-14T16:07:05.832899 webserver_plugin.cpp:658 plugin_pre_shutdown ] Shutting down webserver_plugin...
2023-02-14T16:07:05.833576 data_processor.cpp:125 trigger ] Trying to trigger data processor: hive.set_irreversible caller but its execution is already canceled. The data are ignored.
2023-02-14T16:07:05.833629 chain_plugin.cpp:496 operator() ] Write processing thread finished.
2023-02-14T16:07:18.972528 webserver_plugin.cpp:276 operator() ] ws io service exit