Mirrornet node receiving API traffic from bridge crashed in broadcast_trx during shutdown
In the past, API nodes shutdown cleanly after ctrl-c, so this issue needs to be resolved. The notable difference from other instances of this version of the code is this one was receiving broadcast_api traffic calls from mirrornet bridge (1/2 of all traffic). The problem occurred on server s6. Another side point is for unknown reasons this node was also eating up a lot of virtual memory from early on 93G, IIRC. But I don't think that is related to the issue.
2022-08-13T21:27:50.284044 p2p_plugin.cpp:583 broadcast_transactio ] Broadcasting tx #5e8d45d4e4842159aa20c4c22aa2c723498fb530
^C2022-08-13T21:27:50.433390 application.cpp:93 handle_signal ] _last_signal_code: 2
2022-08-13T21:27:50.433423 application.cpp:84 generate_interrupt_r ] interrupt requested!
2022-08-13T21:27:50.433443 p2p_plugin.cpp:548 plugin_pre_shutdown ] Shutting down P2P Plugin...
2022-08-13T21:27:50.434042 shutdown_mgr.hpp:56 wait ] Processing of 'shutdown-state type: HIVE_P2P_BLOCK_HANDLER' in progress...
2022-08-13T21:27:50.434052 shutdown_mgr.hpp:72 wait ] A value from a different thread is read...
2022-08-13T21:27:50.434059 shutdown_mgr.hpp:77 wait ] Processing of 'shutdown-state type: HIVE_P2P_BLOCK_HANDLER' was finished...
2022-08-13T21:27:50.434064 shutdown_mgr.hpp:56 wait ] Processing of 'shutdown-state type: HIVE_P2P_TRANSACTION_HANDLER' in progress...
2022-08-13T21:27:50.434068 shutdown_mgr.hpp:72 wait ] A value from a different thread is read...
2022-08-13T21:27:50.434073 shutdown_mgr.hpp:77 wait ] Processing of 'shutdown-state type: HIVE_P2P_TRANSACTION_HANDLER' was finished...
2022-08-13T21:27:50.434077 p2p_plugin.cpp:558 plugin_pre_shutdown ] P2P Plugin: checking handle_block and handle_transaction activity
2022-08-13T21:27:50.468168 p2p_plugin.cpp:562 plugin_pre_shutdown ] Waiting for p2p_thread quit
2022-08-13T21:27:50.468183 p2p_plugin.cpp:564 plugin_pre_shutdown ] p2p_thread quit done
2022-08-13T21:27:50.972477 state_snapshot_plugin.cpp:1471 plugin_shutdown ] Shutting down state_snapshot_plugin...
2022-08-13T21:27:50.973253 webserver_plugin.cpp:273 operator() ] http io service exit
2022-08-13T21:27:50.973425 webserver_plugin.cpp:234 operator() ] ws io service exit
2022-08-13T21:27:51.265578 p2p_plugin.cpp:583 broadcast_transactio ] Broadcasting tx #04f7995a9f3ca63a73e2ff42960ad82a21587763
Before shutting down...
Shutting down...
stack trace:
0x00005609d651a609 /storage1/hived-logging/build/programs/hived/hived : fc::segfault_handler(int, siginfo_t*, void*)+0x29
0x00007effdff9b520 /lib/x86_64-linux-gnu/libc.so.6 : +0x42520
0x00005609d57e206b /storage1/hived-logging/build/programs/hived/hived : hive::plugins::p2p::p2p_plugin::broadcast_transaction(std::shared_ptr<hive::chain::full_transaction_type> const&)+0xbb
0x00005609d4ed2ac6 /storage1/hived-logging/build/programs/hived/hived : hive::plugins::network_broadcast_api::detail::network_broadcast_api_impl::broadcast_transaction(hive::plugins::network_broadcast_api::broadcast_transaction_args const&)+0xd6
0x00005609d4ed2bd0 /storage1/hived-logging/build/programs/hived/hived : hive::plugins::network_broadcast_api::network_broadcast_api::broadcast_transaction(hive::plugins::network_broadcast_api::broadcast_transaction_args const&, bool)+0x10
0x00005609d4ee034a /storage1/hived-logging/build/programs/hived/hived : std::_Function_handler<fc::variant (fc::variant const&), hive::plugins::json_rpc::detail::register_api_method_visitor::operator()<hive::plugins::network_broadcast_api::network_broadcast_api, hive::plugins::json_rpc::void_type (hive::plugins::network_broadcast_api::network_broadcast_api::*)(hive::plugins::network_broadcast_api::broadcast_transaction_args const&, bool), hive::plugins::network_broadcast_api::broadcast_transaction_args, hive::plugins::json_rpc::void_type>(hive::plugins::network_broadcast_api::network_broadcast_api&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, hive::plugins::json_rpc::void_type (hive::plugins::network_broadcast_api::network_broadcast_api::*)(hive::plugins::network_broadcast_api::broadcast_transaction_args const&, bool), hive::plugins::network_broadcast_api::broadcast_transaction_args*, hive::plugins::json_rpc::void_type*)::{lambda(fc::variant const&)#1}>::_M_invoke(std::_Any_data const&, fc::variant const&)+0x23a
0x00005609d5704aff /storage1/hived-logging/build/programs/hived/hived : hive::plugins::json_rpc::detail::json_rpc_plugin_impl::rpc_jsonrpc(fc::variant_object const&, hive::plugins::json_rpc::detail::json_rpc_response&)+0x61f
0x00005609d57054c0 /storage1/hived-logging/build/programs/hived/hived : hive::plugins::json_rpc::detail::json_rpc_plugin_impl::rpc(fc::variant const&)+0x4a0
0x00005609d5707164 /storage1/hived-logging/build/programs/hived/hived : hive::plugins::json_rpc::json_rpc_plugin::call(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x454
0x00005609d55cc2ce /storage1/hived-logging/build/programs/hived/hived : +0x1ba72ce
0x00005609d55cce14 /storage1/hived-logging/build/programs/hived/hived : +0x1ba7e14
0x00005609d4bcc1b3 /storage1/hived-logging/build/programs/hived/hived : boost::asio::detail::scheduler::run(boost::system::error_code&)+0x4f3
0x00005609d55c43fa /storage1/hived-logging/build/programs/hived/hived : +0x1b9f3fa
0x00005609d682f45b /storage1/hived-logging/build/programs/hived/hived : +0x2e0a45b
0x00007effdffedb43 /lib/x86_64-linux-gnu/libc.so.6 : +0x94b43
0x00007effe007fa00 /lib/x86_64-linux-gnu/libc.so.6 : +0x126a00
executing command(s):
addr2line -p -a -f -C -i -e /storage1/hived-logging/build/programs/hived/hived +0x29
addr2line -p -a -f -C -i -e /lib/x86_64-linux-gnu/libc.so.6 +0x42520
addr2line -p -a -f -C -i -e /storage1/hived-logging/build/programs/hived/hived +0xbb +0xd6 +0x10 +0x23a +0x61f +0x4a0 +0x454 +0x1ba72ce +0x1ba7e14 +0x4f3 +0x1b9f3fa +0x2e0a45b
addr2line -p -a -f -C -i -e /lib/x86_64-linux-gnu/libc.so.6 +0x94b43 +0x126a00
0x0000000000000029: ?? ??:0
0x0000000000042520: __restore_rt at libc_sigaction.c:?
0x00000000000000bb: ?? ??:0
0x00000000000000d6: ?? ??:0
0x0000000000000010: ?? ??:0
0x000000000000023a: ?? ??:0
0x000000000000061f: ?? ??:0
0x00000000000004a0: ?? ??:0
0x0000000000000454: ?? ??:0
0x0000000001ba72ce: _ZN4hive5chain15_add_index_implIN5boost11multi_index21multi_index_containerINS_7plugins6follow13follow_objectENS3_10indexed_byINS3_14ordered_uniqueINS3_3tagIN9chainbase5by_idEN4mpl_2naESE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_EENS3_13const_mem_funIS7_NSB_3oidIS7_EEXadL_ZNKS7_6get_idEvEEEESE_EENS9_INSA_INS6_21by_following_followerESE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_EENS3_13composite_keyIS7_NS3_6memberIS7_NS_8protocol17fixed_string_implIN2fc7uint128EEEXadL_ZNS7_9followingEEEEENSO_IS7_ST_XadL_ZNS7_8followerEEEEENS2_6tuples9null_typeESX_SX_SX_SX_SX_SX_SX_EENS3_21composite_key_compareISt4lessIST_ES11_SX_SX_SX_SX_SX_SX_SX_SX_EEEENS9_INSA_INS6_21by_follower_followingESE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_EENSN_IS7_SV_SU_SX_SX_SX_SX_SX_SX_SX_SX_EES12_EESE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_EENS2_12interprocess9allocatorIS7_NS19_15segment_managerIcNS19_15rbtree_best_fitINS19_12mutex_familyENS19_10offset_ptrIvlmLm0EEELm0EEENS19_10iset_indexEEEEEEEEEvRNS0_8databaseE at ??:?
0x0000000001ba7e14: _ZN4hive5chain15_add_index_implIN5boost11multi_index21multi_index_containerINS_7plugins6follow11blog_objectENS3_10indexed_byINS3_14ordered_uniqueINS3_3tagIN9chainbase5by_idEN4mpl_2naESE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_EENS3_13const_mem_funIS7_NSB_3oidIS7_EEXadL_ZNKS7_6get_idEvEEEESE_EENS9_INSA_INS6_7by_blogESE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_EENS3_13composite_keyIS7_NS3_6memberIS7_NS_8protocol17fixed_string_implIN2fc7uint128EEEXadL_ZNS7_7accountEEEEENSO_IS7_jXadL_ZNS7_12blog_feed_idEEEEENS2_6tuples9null_typeESX_SX_SX_SX_SX_SX_SX_EENS3_21composite_key_compareISt4lessIST_ESt7greaterIjESX_SX_SX_SX_SX_SX_SX_SX_EEEENS9_INSA_INS6_10by_commentESE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_EENSN_IS7_NSO_IS7_NSB_7oid_refINS0_14comment_objectEEEXadL_ZNS7_7commentEEEEESU_SJ_SX_SX_SX_SX_SX_SX_SX_EENSZ_IS10_IS1A_ES11_S10_INS18_IS7_EEESX_SX_SX_SX_SX_SX_SX_EEEESE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_EENS2_12interprocess9allocatorIS7_NS1J_15segment_managerIcNS1J_15rbtree_best_fitINS1J_12mutex_familyENS1J_10offset_ptrIvlmLm0EEELm0EEENS1J_10iset_indexEEEEEEEEEvRNS0_8databaseE at ??:?
0x00000000000004f3: ?? ??:0
0x0000000001b9f3fa: _ZNK9chainbase29generic_index_snapshot_dumperINS_13generic_indexIN5boost11multi_index21multi_index_containerIN4hive7plugins6follow11feed_objectENS3_10indexed_byINS3_14ordered_uniqueINS3_3tagINS_5by_idEN4mpl_2naESE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_EENS3_13const_mem_funIS8_NS_3oidIS8_EEXadL_ZNKS8_6get_idEvEEEESE_EENSA_INSB_INS7_7by_feedESE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_EENS3_13composite_keyIS8_NS3_6memberIS8_NS5_8protocol17fixed_string_implIN2fc7uint128EEEXadL_ZNS8_7accountEEEEENSO_IS8_jXadL_ZNS8_15account_feed_idEEEEENS2_6tuples9null_typeESX_SX_SX_SX_SX_SX_SX_EENS3_21composite_key_compareISt4lessIST_ESt7greaterIjESX_SX_SX_SX_SX_SX_SX_SX_EEEENSA_INSB_INS7_10by_commentESE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_EENSN_IS8_NSO_IS8_NS_7oid_refINS5_5chain14comment_objectEEEXadL_ZNS8_7commentEEEEESU_SJ_SX_SX_SX_SX_SX_SX_SX_EENSZ_IS10_IS1B_ES11_S10_INS18_IS8_EEESX_SX_SX_SX_SX_SX_SX_EEEESE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_EENS2_12interprocess9allocatorIS8_NS1K_15segment_managerIcNS1K_15rbtree_best_fitINS1K_12mutex_familyENS1K_10offset_ptrIvlmLm0EEELm0EEENS1K_10iset_indexEEEEEEEEEE11dumper_dataIS1V_E12doConversionEPNS_15snapshot_writer6workerE at ??:?
0x0000000002e0a45b: v2i_IPAddrBlocks at libcrypto-lib-v3_addr.o:?
0x0000000000094b43: start_thread at ./nptl/pthread_create.c:442
0x0000000000126a00: __clone3 at ./misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:83
terminate called without an active exception
Aborted