hived failed to shutdown cleanly on ^C because unprocessed blocks remained in the queue
hived (running with haf) was sent a SIGINT and attempted to shut down gracefully. It began the shutdown procedure, but during p2p plugin shutdown it failed to process all blocks in the queue in the 30 seconds allotted, and then aborted.
The relevant part of the log:
2023-09-13T20:33:44.026609 chain_plugin.cpp:1155 connection_count_cha ] peer_count changed: 19
2023-09-13T20:36:17.446593 chain_plugin.cpp:1155 connection_count_cha ] peer_count changed: 20
2023-09-13T20:36:18.043819 chain_plugin.cpp:1182 accept_block ] Syncing Blockchain --- Got block: #78380000 time: 2023-09-13T20:34:12 producer: deathwing
Performing cleanup....
Hived pid: 156
[1]+ 152 Running { sudo --user=hived -En /bin/bash <<EOF
echo "Attempting to execute hived using additional command line arguments:" "${HIVED_ARGS[@]}"
/home/hived/bin/hived --webserver-ws-endpoint=0.0.0.0:${WS_PORT} --webserver-http-endpoint=0.0.0.0:${HTTP_PORT} --p2p-endpoint=0.0.0.0:${P2P_PORT} --data-dir="$DATADIR" --shared-file-dir="$SHM_DIR" --plugin=sql_serializer --psql-url="dbname=haf_block_log host=/var/run/postgresql port=5432" ${HIVED_ARGS[@]} 2>&1 | tee -i hived.log
echo "$? Hived process finished execution."
EOF
stop_postresql; } &
2023-09-13T20:37:54.662816 application.cpp:99 handle_signal ] _last_signal_code: 2
2023-09-13T20:37:54.663209 application.cpp:90 generate_interrupt_r ] interrupt requested!
2023-09-13T20:37:54.663698 webserver_plugin.cpp:651 plugin_pre_shutdown ] Shutting down webserver_plugin...
Waiting for hived finish...
2023-09-13T20:37:54.664728 webserver_plugin.cpp:277 operator() ] ws io service exit
2023-09-13T20:37:54.665498 webserver_plugin.cpp:310 operator() ] http io service exit
2023-09-13T20:37:54.665800 p2p_plugin.cpp:545 plugin_pre_shutdown ] Shutting down P2P Plugin...
2023-09-13T20:37:54.666103 shutdown_mgr.hpp:96 wait ] Processing of 'P2P_BLOCK' in progress...
2023-09-13T20:37:54.766185 shutdown_mgr.hpp:105 wait ] attempt: 1/300, reason: timeout, future status(internal): 1 ...
2023-09-13T20:37:54.766229 shutdown_mgr.hpp:106 wait ] Details: Wait for: P2P_BLOCK. Currently 73 of 'P2P_BLOCK' items are processed...
2023-09-13T20:37:54.866312 shutdown_mgr.hpp:105 wait ] attempt: 2/300, reason: timeout, future status(internal): 1 ...
2023-09-13T20:37:54.866360 shutdown_mgr.hpp:106 wait ] Details: Wait for: P2P_BLOCK. Currently 73 of 'P2P_BLOCK' items are processed...
2023-09-13T20:37:54.966441 shutdown_mgr.hpp:105 wait ] attempt: 3/300, reason: timeout, future status(internal): 1 ...
2023-09-13T20:37:54.966488 shutdown_mgr.hpp:106 wait ] Details: Wait for: P2P_BLOCK. Currently 73 of 'P2P_BLOCK' items are processed...
2023-09-13T20:37:55.066568 shutdown_mgr.hpp:105 wait ] attempt: 4/300, reason: timeout, future status(internal): 1 ...
2023-09-13T20:37:55.066608 shutdown_mgr.hpp:106 wait ] Details: Wait for: P2P_BLOCK. Currently 73 of 'P2P_BLOCK' items are processed...
2023-09-13T20:37:55.166692 shutdown_mgr.hpp:105 wait ] attempt: 5/300, reason: timeout, future status(internal): 1 ...
2023-09-13T20:37:55.166744 shutdown_mgr.hpp:106 wait ] Details: Wait for: P2P_BLOCK. Currently 73 of 'P2P_BLOCK' items are processed...
2023-09-13T20:37:55.266844 shutdown_mgr.hpp:105 wait ] attempt: 6/300, reason: timeout, future status(internal): 1 ...
2023-09-13T20:37:55.266912 shutdown_mgr.hpp:106 wait ] Details: Wait for: P2P_BLOCK. Currently 73 of 'P2P_BLOCK' items are processed...
2023-09-13T20:37:55.367007 shutdown_mgr.hpp:105 wait ] attempt: 7/300, reason: timeout, future status(internal): 1 ...
2023-09-13T20:37:55.367057 shutdown_mgr.hpp:106 wait ] Details: Wait for: P2P_BLOCK. Currently 73 of 'P2P_BLOCK' items are processed...
2023-09-13T20:37:55.467148 shutdown_mgr.hpp:105 wait ] attempt: 8/300, reason: timeout, future status(internal): 1 ...
2023-09-13T20:37:55.467191 shutdown_mgr.hpp:106 wait ] Details: Wait for: P2P_BLOCK. Currently 73 of 'P2P_BLOCK' items are processed...
2023-09-13T20:37:55.567275 shutdown_mgr.hpp:105 wait ] attempt: 9/300, reason: timeout, future status(internal): 1 ...
2023-09-13T20:37:55.567322 shutdown_mgr.hpp:106 wait ] Details: Wait for: P2P_BLOCK. Currently 73 of 'P2P_BLOCK' items are processed...
.... quite a few omitted ....
2023-09-13T20:38:24.506561 shutdown_mgr.hpp:105 wait ] attempt: 298/300, reason: timeout, future status(internal): 1 ...
2023-09-13T20:38:24.506592 shutdown_mgr.hpp:106 wait ] Details: Wait for: P2P_BLOCK. Currently 73 of 'P2P_BLOCK' items are processed...
2023-09-13T20:38:24.606686 shutdown_mgr.hpp:105 wait ] attempt: 299/300, reason: timeout, future status(internal): 1 ...
2023-09-13T20:38:24.606717 shutdown_mgr.hpp:106 wait ] Details: Wait for: P2P_BLOCK. Currently 73 of 'P2P_BLOCK' items are processed...
2023-09-13T20:38:24.706812 shutdown_mgr.hpp:105 wait ] attempt: 300/300, reason: timeout, future status(internal): 1 ...
2023-09-13T20:38:24.706843 shutdown_mgr.hpp:106 wait ] Details: Wait for: P2P_BLOCK. Currently 73 of 'P2P_BLOCK' items are processed...
2023-09-13T20:38:24.806921 shutdown_mgr.hpp:105 wait ] attempt: 301/300, reason: timeout, future status(internal): 1 ...
2023-09-13T20:38:24.806952 shutdown_mgr.hpp:106 wait ] Details: Wait for: P2P_BLOCK. Currently 73 of 'P2P_BLOCK' items are processed...
Executing `pre shutdown` for all plugins...
Before shutting down...
Plugin: p2p raised an exception...
10 assert_exception: Assert Exception
++cnt <= time_maximum
Closing the P2P plugin is terminated
{"name":"P2P plugin"}
shutdown_mgr.hpp:108 wait
terminate called without an active exception
1 Hived process finished execution.
Attempting to stop Postgresql...
* Stopping PostgreSQL 14 database server
Hived finish done.
Attempting to stop Postgresql...
* Stopping PostgreSQL 14 database server
...done.
Waiting for postgres process: 69 finish...
Postgres process: 69 finished.
...done.
Waiting for postgres process: 69 finish...
Postgres process: 69 finished.
Cleanup actions done.