Shutdown issues (p2p active even after write processing thread finished)
After I attempted to shut the node down, it actually gains more p2p connections, and continue doing so even after "Write processing thread finished."
2022-10-21T08:08:57.402 p2p_plugin.cpp:242 handle_transaction ] Transaction ignored due to start p2p_plugin shutdown
2022-10-21T08:08:57.402 p2p_plugin.cpp:242 handle_transaction ] Transaction ignored due to start p2p_plugin shutdown
2022-10-21T08:08:57.585 chain_plugin.cpp:1016 connection_count_cha ] peer_count changed: 3
2022-10-21T08:08:57.585 chain_plugin.cpp:1016 connection_count_cha ] peer_count changed: 4
2022-10-21T08:08:57.586 chain_plugin.cpp:1016 connection_count_cha ] peer_count changed: 5
2022-10-21T08:08:57.675 p2p_plugin.cpp:242 handle_transaction ] Transaction ignored due to start p2p_plugin shutdown
2022-10-21T08:08:57.675 p2p_plugin.cpp:242 handle_transaction ] Transaction ignored due to start p2p_plugin shutdown
2022-10-21T08:08:57.675 p2p_plugin.cpp:242 handle_transaction ] Transaction ignored due to start p2p_plugin shutdown
2022-10-21T08:08:57.675 p2p_plugin.cpp:242 handle_transaction ] Transaction ignored due to start p2p_plugin shutdown
2022-10-21T08:08:57.675 p2p_plugin.cpp:242 handle_transaction ] Transaction ignored due to start p2p_plugin shutdown
2022-10-21T08:08:57.675 p2p_plugin.cpp:242 handle_transaction ] Transaction ignored due to start p2p_plugin shutdown
2022-10-21T08:08:57.675 p2p_plugin.cpp:242 handle_transaction ] Transaction ignored due to start p2p_plugin shutdown
2022-10-21T08:08:57.675 p2p_plugin.cpp:242 handle_transaction ] Transaction ignored due to start p2p_plugin shutdown
2022-10-21T08:08:57.675 p2p_plugin.cpp:242 handle_transaction ] Transaction ignored due to start p2p_plugin shutdown
2022-10-21T08:08:57.675 p2p_plugin.cpp:242 handle_transaction ] Transaction ignored due to start p2p_plugin shutdown
2022-10-21T08:08:57.675 p2p_plugin.cpp:242 handle_transaction ] Transaction ignored due to start p2p_plugin shutdown
2022-10-21T08:08:57.676 p2p_plugin.cpp:242 handle_transaction ] Transaction ignored due to start p2p_plugin shutdown
2022-10-21T08:08:57.676 p2p_plugin.cpp:242 handle_transaction ] Transaction ignored due to start p2p_plugin shutdown
2022-10-21T08:08:57.676 p2p_plugin.cpp:242 handle_transaction ] Transaction ignored due to start p2p_plugin shutdown
2022-10-21T08:08:57.676 p2p_plugin.cpp:242 handle_transaction ] Transaction ignored due to start p2p_plugin shutdown
2022-10-21T08:08:57.676 p2p_plugin.cpp:242 handle_transaction ] Transaction ignored due to start p2p_plugin shutdown
2022-10-21T08:08:57.676 p2p_plugin.cpp:242 handle_transaction ] Transaction ignored due to start p2p_plugin shutdown
2022-10-21T08:08:57.676 p2p_plugin.cpp:242 handle_transaction ] Transaction ignored due to start p2p_plugin shutdown
2022-10-21T08:08:57.676 p2p_plugin.cpp:242 handle_transaction ] Transaction ignored due to start p2p_plugin shutdown
2022-10-21T08:08:57.676 p2p_plugin.cpp:242 handle_transaction ] Transaction ignored due to start p2p_plugin shutdown
2022-10-21T08:08:57.676 p2p_plugin.cpp:242 handle_transaction ] Transaction ignored due to start p2p_plugin shutdown
2022-10-21T08:08:57.676 p2p_plugin.cpp:242 handle_transaction ] Transaction ignored due to start p2p_plugin shutdown
2022-10-21T08:08:57.676 p2p_plugin.cpp:242 handle_transaction ] Transaction ignored due to start p2p_plugin shutdown
2022-10-21T08:08:57.676 p2p_plugin.cpp:242 handle_transaction ] Transaction ignored due to start p2p_plugin shutdown
2022-10-21T08:08:57.681 chain_plugin.cpp:1016 connection_count_cha ] peer_count changed: 6
2022-10-21T08:08:57.682 chain_plugin.cpp:1016 connection_count_cha ] peer_count changed: 7
2022-10-21T08:08:57.682 chain_plugin.cpp:1016 connection_count_cha ] peer_count changed: 8
2022-10-21T08:08:57.683 chain_plugin.cpp:1016 connection_count_cha ] peer_count changed: 9
2022-10-21T08:08:57.684 chain_plugin.cpp:1016 connection_count_cha ] peer_count changed: 10
2022-10-21T08:08:57.684 chain_plugin.cpp:1016 connection_count_cha ] peer_count changed: 11
2022-10-21T08:08:57.701 chain_plugin.cpp:1016 connection_count_cha ] peer_count changed: 12
2022-10-21T08:08:57.710 p2p_plugin.cpp:242 handle_transaction ] Transaction ignored due to start p2p_plugin shutdown
2022-10-21T08:08:57.710 p2p_plugin.cpp:242 handle_transaction ] Transaction ignored due to start p2p_plugin shutdown
2022-10-21T08:08:57.710 p2p_plugin.cpp:242 handle_transaction ] Transaction ignored due to start p2p_plugin shutdown
2022-10-21T08:08:57.710 p2p_plugin.cpp:242 handle_transaction ] Transaction ignored due to start p2p_plugin shutdown
2022-10-21T08:08:57.710 p2p_plugin.cpp:242 handle_transaction ] Transaction ignored due to start p2p_plugin shutdown
2022-10-21T08:08:57.711 p2p_plugin.cpp:242 handle_transaction ] Transaction ignored due to start p2p_plugin shutdown
2022-10-21T08:08:57.711 p2p_plugin.cpp:242 handle_transaction ] Transaction ignored due to start p2p_plugin shutdown
2022-10-21T08:08:57.711 p2p_plugin.cpp:242 handle_transaction ] Transaction ignored due to start p2p_plugin shutdown
2022-10-21T08:08:57.720 chain_plugin.cpp:1016 connection_count_cha ] peer_count changed: 13
2022-10-21T08:09:05.048 chain_plugin.cpp:1016 connection_count_cha ] peer_count changed: 14
2022-10-21T08:09:06.725 chain_plugin.cpp:1016 connection_count_cha ] peer_count changed: 15
2022-10-21T08:09:07.586 chain_plugin.cpp:1016 connection_count_cha ] peer_count changed: 16
2022-10-21T08:09:07.709 chain_plugin.cpp:1016 connection_count_cha ] peer_count changed: 17
2022-10-21T08:09:09.326 chain_plugin.cpp:496 operator() ] Write processing thread finished.
^C^C^C
and it's not exiting (most likely) because of ESTABLISHED p2p connections doesn't respond to further Ctrl+C
^C^C^C2022-10-21T08:13:07.237 chain_plugin.cpp:1016 connection_count_cha ] peer_count changed: 18
^C^C^C2022-10-21T08:14:05.723 chain_plugin.cpp:1016 connection_count_cha ] peer_count changed: 19
^C^C^C2022-10-21T08:14:21.299 chain_plugin.cpp:1016 connection_count_cha ] peer_count changed: 18
and continue to connect to the outside world
I assumed that because of "Write proessing thread finished" it's in consistent state and can be forcibly terminated without negative consequences, and that's not really that simple because restart complained:
The transaction status plugin state does not contain tracking information for the last 64000 blocks. Re-run with the command line argument '--transaction-status-rebuild-state'.
Here's more on that:
2022-10-21T08:30:23.141 block_log.cpp:158 open ] Opening blocklog /home/hive/datadir-ah/blockchain/block_log
2022-10-21T08:30:23.141 block_log.cpp:187 open ] block_log_size: 368617208722
2022-10-21T08:30:23.142 block_log_artifacts.cpp:386 load_header ] Loaded header containing: git rev: 64f7389128f3c1a59f2c8107509d5f6de37aa2d9, format version: 1.0, head_block_num: 68651023
2022-10-21T08:30:23.142 block_log_artifacts.cpp:334 try_to_open ] block_log file is longer than current block_log.artifact file - artifact file generation will be resumed for range: <68651024:68959230>.
2022-10-21T08:30:23.142 block_log_artifacts.cpp:422 generate_file ] Attempting to generate a block artifact file for block range: 68651024...68959230
2022-10-21T08:30:23.142 block_log.cpp:741 for_each_block ] Walking over block log starting from block: 68959230...
2022-10-21T08:30:28.430 block_log.cpp:767 for_each_block ] Stopping block position list walk on caller request... Last processed block: 68651023
2022-10-21T08:30:28.550 block_log.cpp:781 for_each_block ] Block log walk finished in time: 5 s.
2022-10-21T08:30:28.551 block_log_artifacts.cpp:563 generate_file ] Block artifact file generation finished. 308207 blocks processed in time: 5409 ms
2022-10-21T08:30:28.551 database.cpp:200 load_state_initial_d ] Loaded a blockchain database holding a state specific to head block: 68959230 and last irreversible block: 68959230
2022-10-21T08:30:28.551 database.cpp:204 operator() ] Attempting to rewind all undo state...
2022-10-21T08:30:28.557 database.cpp:208 operator() ] Rewind undo state done.
2022-10-21T08:30:28.557 database.cpp:218 operator() ] Blockchain state database is AT IRREVERSIBLE state specific to head block: 68959230 and LIB: 68959230
2022-10-21T08:31:42.646 database.cpp:7055 validate_invariants ] validate_invariants @68959230:
2022-10-21T08:31:42.646 database.cpp:7056 validate_invariants ] successful scan of 14964 witnesses, 2426215 accounts, 1067 convert requests, 56 collateralized convert requests, 990 limit orders, 32 escrow transfers, 111 saving withdrawals, 1 reward funds and 0 SMT contributions.
2022-10-21T08:31:42.646 database.cpp:7058 validate_invariants ] HIVE supply: 388217170938
2022-10-21T08:31:42.646 database.cpp:7059 validate_invariants ] HBD supply: 28291630306 ( + 0 initial )
2022-10-21T08:31:42.646 database.cpp:7060 validate_invariants ] virtual supply (HIVE): 448412129035
2022-10-21T08:31:42.646 database.cpp:7061 validate_invariants ] VESTS: 296756719235589095 ( + 835577875731178 pending ) worth (HIVE): 164183632696 ( + 434817532 )
2022-10-21T08:31:42.646 database.cpp:258 operator() ] Loaded blockchain which had already processed hardfork 24, setting Hive chain id
2022-10-21T08:31:42.646 database.cpp:665 set_chain_id ] hive_chain_id: beeab0de00000000000000000000000000000000000000000000000000000000
2022-10-21T08:31:42.646 chain_plugin.cpp:953 plugin_startup ] Looking for snapshot processing requests...
2022-10-21T08:31:42.646 chain_plugin.cpp:968 plugin_startup ] Consistency data checking...
2022-10-21T08:31:42.646 chain_plugin.cpp:983 plugin_startup ] P2P enabling...
2022-10-21T08:31:42.646 chain_plugin.cpp:721 work ] Started on blockchain with 68959230 blocks, LIB: 68959230
2022-10-21T08:31:42.646 chain_plugin.cpp:727 work ] Started on blockchain with 68959230 blocks
2022-10-21T08:31:42.647 chain_plugin.cpp:989 plugin_startup ] Chain plugin initialization finished...
2022-10-21T08:31:42.647 chain_plugin.cpp:339 operator() ] Write processing thread started.
2022-10-21T08:31:42.647 p2p_plugin.cpp:464 plugin_startup ] P2P plugin startup...
(...)
2022-10-21T08:31:42.655 p2p_plugin.cpp:528 operator() ] P2P node listening at 0.0.0.0:2001
2022-10-21T08:31:42.658 chain_plugin.cpp:1016 connection_count_cha ] peer_count changed: 1
2022-10-21T08:31:42.675 transaction_status_plugin.cpp:111 on_post_apply_block ] Transaction status tracking activated at block 68959231, statuses will be available after block 68400000
2022-10-21T08:31:42.675 database.cpp:4242 check_free_memory ] Free memory is now 10G. Current block number: 68959231
2022-10-21T08:31:42.678 p2p_plugin.cpp:537 plugin_startup ] P2P Plugin started
2022-10-21T08:31:42.678 witness_plugin.cpp:648 plugin_startup ] witness plugin: plugin_startup() begin
2022-10-21T08:31:42.678 witness_plugin.cpp:671 plugin_startup ] No witnesses configured! Please add witness IDs and private keys to configuration.
2022-10-21T08:31:42.678 witness_plugin.cpp:672 plugin_startup ] witness plugin: plugin_startup() end
2022-10-21T08:31:42.678 account_history_rocksdb_plugin.cpp:2204 plugin_startup ] Starting up account_history_rocksdb_plugin...
2022-10-21T08:31:42.678 transaction_status_plugin.cpp:307 plugin_startup ] transaction_status: plugin_startup() begin
2022-10-21T08:31:42.678 transaction_status_plugin.cpp:317 plugin_startup ] The transaction status plugin state does not contain tracking information for the last 64000 blocks. Re-run with the command line argument '--transaction-status-rebuild-state'.
terminate called without an active exception
and yet, starting it again worked fine, even without --transaction-status-update
as I guess those extra blocks in block_log were replayed and transaction status updated.