[mirror-net] crash on shutdown
While trying to stop a node in a #350 (closed) condition, I got this:
2022-08-10T16:44:26.755 shutdown_mgr.hpp:65 wait ] finishing: timeout, future status: 1, attempt: 297 ...
2022-08-10T16:44:26.855 shutdown_mgr.hpp:65 wait ] finishing: timeout, future status: 1, attempt: 298 ...
2022-08-10T16:44:26.955 shutdown_mgr.hpp:65 wait ] finishing: timeout, future status: 1, attempt: 299 ...
2022-08-10T16:44:27.055 shutdown_mgr.hpp:65 wait ] finishing: timeout, future status: 1, attempt: 300 ...
2022-08-10T16:44:27.155 shutdown_mgr.hpp:65 wait ] finishing: timeout, future status: 1, attempt: 301 ...
Before shutting down...
application shutdown: unknown error exception.
Leaving application main loop...
2022-08-10T16:44:29.260 db_with.hpp:166 ~pending_transaction ] Postponed 2116724471 pending transactions. 0 were applied. 0 expired.
2022-08-10T16:44:34.538 chain_plugin.cpp:419 operator() ] Stopped processing write_queue before empty because we exceeded 500ms, held lock for 18904862μs
2022-08-10T16:44:34.539 chain_plugin.cpp:473 operator() ] waiting for work: 0.00%, waiting for locks: 0.00%, processing transactions: 0.00%, processing blocks: 0.00%, unknown: 100.00%
2022-08-10T16:44:34.539 database.cpp:1078 switch_forks ] Switching to fork: 03fc2c6095d30a058082e893b5ccd9e24372b3d7
2022-08-10T16:44:34.572 webserver_plugin.cpp:468 operator() ] Excessive delay to process API call: 72.631 s
2022-08-10T16:44:34.572 webserver_plugin.cpp:468 operator() ] Excessive delay to process API call: 42.400 s
2022-08-10T16:44:34.572 webserver_plugin.cpp:468 operator() ] Excessive delay to process API call: 11.969 s
2022-08-10T16:44:35.062 webserver_plugin.cpp:468 operator() ] Excessive delay to process API call: 23.961 s
2022-08-10T16:44:35.062 webserver_plugin.cpp:468 operator() ] Excessive delay to process API call: 54.192 s
2022-08-10T16:44:35.416 webserver_plugin.cpp:468 operator() ] Excessive delay to process API call: 73.208 s
2022-08-10T16:44:35.872 webserver_plugin.cpp:468 operator() ] Excessive delay to process API call: 43.314 s
2022-08-10T16:44:35.872 webserver_plugin.cpp:468 operator() ] Excessive delay to process API call: 12.883 s
2022-08-10T16:44:35.872 webserver_plugin.cpp:468 operator() ] Excessive delay to process API call: 73.545 s
2022-08-10T16:44:36.235 chain_plugin.cpp:997 connection_count_cha ] peer_count changed: 0
2022-08-10T16:44:36.989 chain_plugin.cpp:997 connection_count_cha ] peer_count changed: 1
exited cleanly
stack trace:
0x0000563bf0f2f889 ./hived-mirror-develop : fc::segfault_handler(int, siginfo_t*, void*)+0x29
0x00007ff2a8395520 /lib/x86_64-linux-gnu/libc.so.6 : +0x42520
0x0000563bf0d74cf3 ./hived-mirror-develop : hive::chain::full_transaction_cache::add_to_cache(std::shared_ptr<hive::chain::full_transaction_type> const&)+0x123
0x0000563bf0d753ef ./hived-mirror-develop : hive::chain::full_transaction_type::create_from_block(std::shared_ptr<hive::chain::decoded_block_storage_type> const&, unsigned int, hive::chain::serialized_transaction_data const&, bool)+0x14f
0x0000563bf0ba6137 ./hived-mirror-develop : hive::chain::full_block_type::decode_block() const+0x377
0x0000563bf0db3181 ./hived-mirror-develop : hive::chain::blockchain_worker_thread_pool::impl::perform_work(std::weak_ptr<hive::chain::full_block_type> const&, hive::chain::blockchain_worker_thread_pool::data_source_type)+0x101
0x0000563bf0db0637 ./hived-mirror-develop : hive::chain::blockchain_worker_thread_pool::impl::thread_function()+0xa7
0x0000563bf0db09ac ./hived-mirror-develop : +0x26a39ac
0x0000563bf1351e64 ./hived-mirror-develop : +0x2c44e64
0x00007ff2a83e7b43 /lib/x86_64-linux-gnu/libc.so.6 : +0x94b43
0x00007ff2a8479a00 /lib/x86_64-linux-gnu/libc.so.6 : +0x126a00
2022-08-10T16:44:37.760 webserver_plugin.cpp:468 operator() ] Excessive delay to process API call: 16.816 s
2022-08-10T16:44:37.763 webserver_plugin.cpp:468 operator() ] Excessive delay to process API call: 47.225 s
2022-08-10T16:44:37.766 webserver_plugin.cpp:468 operator() ] Excessive delay to process API call: 77.458 s
2022-08-10T16:44:37.877 webserver_plugin.cpp:468 operator() ] Excessive delay to process API call: 47.184 s
2022-08-10T16:44:37.877 webserver_plugin.cpp:468 operator() ] Excessive delay to process API call: 16.753 s
2022-08-10T16:44:37.877 webserver_plugin.cpp:468 operator() ] Excessive delay to process API call: 77.415 s
2022-08-10T16:44:38.166 webserver_plugin.cpp:468 operator() ] Excessive delay to process API call: 40.081 s
executing command(s):
addr2line -p -a -f -C -i -e ./hived-mirror-develop +0x29
addr2line -p -a -f -C -i -e /lib/x86_64-linux-gnu/libc.so.6 +0x42520
addr2line -p -a -f -C -i -e ./hived-mirror-develop +0x123 +0x14f +0x377 +0x101 +0xa7 +0x26a39ac +0x2c44e64
addr2line -p -a -f -C -i -e /lib/x86_64-linux-gnu/libc.so.6 +0x94b43 +0x126a00
0x0000000000000029: ?? ??:0
0x0000000000042520: __sigaction at ??:?
0x0000000000000123: ?? ??:0
0x000000000000014f: ?? ??:0
0x0000000000000377: ?? ??:0
0x0000000000000101: ?? ??:0
0x00000000000000a7: ?? ??:0
0x00000000026a39ac: hive::chain::blockchain_worker_thread_pool::blockchain_worker_thread_pool()::{lambda()#1}::operator()() const [clone .isra.0] at blockchain_worker_thread_pool.cpp:?
0x0000000002c44e64: execute_native_thread_routine at thread.o:?
0x0000000000094b43: pthread_condattr_setpshared at ??:?
0x0000000000126a00: __xmknodat at ??:?
terminate called without an active exception