Starting up a node with new version after old version reached HF block failed (hb >= last_irreversible_block)
So I had two similar nodes that were acting as p2p nodes and backups for witnesses.
They were running v1.26.0rc5
and kept doing that until HF27 was applied, and then a little bit longer, staying of course at the obsolete fork.
Then I stopped them, got v1.27.0
binary and attempted to start them again, which failed:
4773ms webserver_plugin.cpp:545 plugin_initialize ] configured with 2 thread pool size
4774ms webserver_plugin.cpp:554 plugin_initialize ] configured http to listen on 127.0.0.1:8091
4774ms webserver_plugin.cpp:571 plugin_initialize ] configured ws to listen on 127.0.0.1:8090
6297ms rc_plugin.cpp:1261 plugin_initialize ] Initializing resource credit plugin
6297ms rc_plugin.cpp:1347 plugin_initialize ] RC's will be computed starting at block 26256743
6297ms witness_plugin.cpp:595 plugin_initialize ] Initializing witness plugin
6298ms state_snapshot_plugin.cpp:1505 plugin_initialize ] Initializing state_snapshot_plugin...
6298ms state_snapshot_plugin.cpp:907 impl ] Registering add_prepare_snapshot_handler...
------------------------------------------------------
@ @@@@@@ ,@@@@@%
@@@@ (@@@@@* @@@@@@
%@@@@@@ @@@@@@ %@@@@@,
@@@@@@@@@@ @@@@@@ @@@@@@
,@@@@@@@@@@@@ @@@@@@ @@@@@@
@@@@@@@@@@@@@@@& @@@@@@ @@@@@@
@@@@@@@@@@@@@@@@@@ .@@@@@% @@@@@@
@@@@@@@@@@@@@@@@@@@@@( .@@@@@%
@@@@@@@@@@@@@@@@@@@@ @@@@@@
*@@@@@@@@@@@@@@@@ @@@@@@ @@@@@@.
@@@@@@@@@@@@@@ &@@@@@. @@@@@@
#@@@@@@@@@@ @@@@@@ #@@@@@/
@@@@@@@@ /@@@@@/ @@@@@@
@@@@@( @@@@@@ .@@@@@&
@@ @@@@@& @@@@@@
STARTING HIVE NETWORK
------------------------------------------------------
initminer public key: STM8GC13uCZbP44HzMLV6zPZGwVQ8Nt4Kji8PapsPiNq1BK153XTX
chain id: 0000000000000000000000000000000000000000000000000000000000000000
blockchain version: 1.27.0
------------------------------------------------------
hived git_revision: "b322c4c19f11f084d09be0dd7d9c615339dce13e"
2022-10-24T13:00:06.300 main.cpp:146 main ] Backtrace on segfault is enabled.
Setting up a startup_io_handler...
2022-10-24T13:00:06.300 chain_plugin.cpp:946 plugin_startup ] Chain plugin initialization...
2022-10-24T13:00:06.300 chain_plugin.cpp:573 initial_settings ] Starting chain with shared_file_size: 25769803776 bytes
2022-10-24T13:00:06.300 chain_plugin.cpp:950 plugin_startup ] Database opening...
2022-10-24T13:00:06.300 chain_plugin.cpp:654 open ] Opening shared memory from /run/hive
Compiler and build environment read from persistent storage: `{"compiler":"9.4.0", "debug":0, "apple":0, "windows":0, "version":{"blockchain_version":"1.26.0","hive_revision":"64f7389128f3c1a59f2c8107509d5f6de37aa2d9","fc_revision":"64f7389128f3c1a59f2c8107509d5f6de37aa2d9"}, "plugins" : ["block_api", "chain", "condenser_api", "database_api", "json_rpc", "network_broadcast_api", "p2p", "rc", "rc_api", "state_snapshot", "webserver", "witness"]}'
2022-10-24T13:00:06.300 database.cpp:147 operator() ] Persistent storage was created according to the version: "version":{"blockchain_version":"1.26.0","hive_revision":"64f7389128f3c1a59f2c8107509d5f6de37aa2d9","fc_revision":"64f7389128f3c1a59f2c8107509d5f6de37aa2d9"} but current node has the version: "version":{"blockchain_version":"1.27.0","hive_revision":"b322c4c19f11f084d09be0dd7d9c615339dce13e","fc_revision":"b322c4c19f11f084d09be0dd7d9c615339dce13e"}
2022-10-24T13:00:06.301 block_log.cpp:158 open ] Opening blocklog /home/hive/.hived/blockchain/block_log
2022-10-24T13:00:06.301 block_log.cpp:187 open ] block_log_size: 383731061986
2022-10-24T13:00:06.303 block_log_artifacts.cpp:386 load_header ] Loaded header containing: git rev: 64f7389128f3c1a59f2c8107509d5f6de37aa2d9, format version: 1.0, head_block_num: 69050083
2022-10-24T13:00:06.303 block_log_artifacts.cpp:345 try_to_open ] block_log and block_log.artifacts files match - no generation needed.
2022-10-24T13:00:06.304 database.cpp:155 open ] 10 assert_exception: Assert Exception
hb >= last_irreversible_block:
{}
database.cpp:198 load_state_initial_data
2022-10-24T13:00:06.304 database.cpp:155 open ] args.data_dir: /home/hive/.hived/blockchain args.shared_mem_dir: /run/hive args.shared_file_size: 25769803776
2022-10-24T13:00:06.305 blockchain_worker_thread_pool.cpp:372 shutdown ] shutting down worker threads
2022-10-24T13:00:06.306 blockchain_worker_thread_pool.cpp:376 shutdown ] worker threads successfully shut down
2022-10-24T13:00:06.306 chain_plugin.cpp:669 open ] Error opening database. If the binary or configuration has changed, replay the blockchain explicitly using `--force-replay`.
2022-10-24T13:00:06.306 chain_plugin.cpp:670 open ] If you know what you are doing you can skip this check and force open the database using `--force-open`.
2022-10-24T13:00:06.306 chain_plugin.cpp:671 open ] WARNING: THIS MAY CORRUPT YOUR DATABASE. FORCE OPEN AT YOUR OWN RISK.
2022-10-24T13:00:06.306 chain_plugin.cpp:672 open ] Error: {"code":10,"name":"assert_exception","message":"Assert Exception","stack":[{"context":{"level":"error","file":"database.cpp","line":198,"method":"load_state_initial_data","hostname":"","timestamp":"2022-10-24T13:00:06"},"format":"hb >= last_irreversible_block: ","data":{}},{"context":{"level":"warn","file":"database.cpp","line":155,"method":"open","hostname":"","timestamp":"2022-10-24T13:00:06"},"format":"rethrow","data":{"args.data_dir":"/home/hive/.hived/blockchain","args.shared_mem_dir":"/run/hive","args.shared_file_size":"25769803776"}}]}
then, trying with --force-open
didn't changed much:
131024ms webserver_plugin.cpp:545 plugin_initialize ] configured with 2 thread pool size
131025ms webserver_plugin.cpp:554 plugin_initialize ] configured http to listen on 127.0.0.1:8091
131025ms webserver_plugin.cpp:571 plugin_initialize ] configured ws to listen on 127.0.0.1:8090
131551ms rc_plugin.cpp:1261 plugin_initialize ] Initializing resource credit plugin
131551ms rc_plugin.cpp:1347 plugin_initialize ] RC's will be computed starting at block 26256743
131551ms witness_plugin.cpp:595 plugin_initialize ] Initializing witness plugin
131552ms state_snapshot_plugin.cpp:1505 plugin_initialize ] Initializing state_snapshot_plugin...
131552ms state_snapshot_plugin.cpp:907 impl ] Registering add_prepare_snapshot_handler...
------------------------------------------------------
@ @@@@@@ ,@@@@@%
@@@@ (@@@@@* @@@@@@
%@@@@@@ @@@@@@ %@@@@@,
@@@@@@@@@@ @@@@@@ @@@@@@
,@@@@@@@@@@@@ @@@@@@ @@@@@@
@@@@@@@@@@@@@@@& @@@@@@ @@@@@@
@@@@@@@@@@@@@@@@@@ .@@@@@% @@@@@@
@@@@@@@@@@@@@@@@@@@@@( .@@@@@%
@@@@@@@@@@@@@@@@@@@@ @@@@@@
*@@@@@@@@@@@@@@@@ @@@@@@ @@@@@@.
@@@@@@@@@@@@@@ &@@@@@. @@@@@@
#@@@@@@@@@@ @@@@@@ #@@@@@/
@@@@@@@@ /@@@@@/ @@@@@@
@@@@@( @@@@@@ .@@@@@&
@@ @@@@@& @@@@@@
STARTING HIVE NETWORK
------------------------------------------------------
initminer public key: STM8GC13uCZbP44HzMLV6zPZGwVQ8Nt4Kji8PapsPiNq1BK153XTX
chain id: 0000000000000000000000000000000000000000000000000000000000000000
blockchain version: 1.27.0
------------------------------------------------------
hived git_revision: "b322c4c19f11f084d09be0dd7d9c615339dce13e"
2022-10-24T13:02:11.553 main.cpp:146 main ] Backtrace on segfault is enabled.
Setting up a startup_io_handler...
2022-10-24T13:02:11.553 chain_plugin.cpp:946 plugin_startup ] Chain plugin initialization...
2022-10-24T13:02:11.554 chain_plugin.cpp:573 initial_settings ] Starting chain with shared_file_size: 25769803776 bytes
2022-10-24T13:02:11.554 chain_plugin.cpp:950 plugin_startup ] Database opening...
2022-10-24T13:02:11.554 chain_plugin.cpp:654 open ] Opening shared memory from /run/hive
2022-10-24T13:02:11.555 block_log.cpp:158 open ] Opening blocklog /home/hive/.hived/blockchain/block_log
2022-10-24T13:02:11.555 block_log.cpp:187 open ] block_log_size: 383731061986
2022-10-24T13:02:11.556 block_log_artifacts.cpp:386 load_header ] Loaded header containing: git rev: 64f7389128f3c1a59f2c8107509d5f6de37aa2d9, format version: 1.0, head_block_num: 69050083
2022-10-24T13:02:11.557 block_log_artifacts.cpp:345 try_to_open ] block_log and block_log.artifacts files match - no generation needed.
2022-10-24T13:02:11.557 database.cpp:155 open ] 10 assert_exception: Assert Exception
hb >= last_irreversible_block:
{}
database.cpp:198 load_state_initial_data
2022-10-24T13:02:11.557 database.cpp:155 open ] args.data_dir: /home/hive/.hived/blockchain args.shared_mem_dir: /run/hive args.shared_file_size: 25769803776
2022-10-24T13:02:11.558 blockchain_worker_thread_pool.cpp:372 shutdown ] shutting down worker threads
2022-10-24T13:02:11.559 blockchain_worker_thread_pool.cpp:376 shutdown ] worker threads successfully shut down
2022-10-24T13:02:11.559 chain_plugin.cpp:669 open ] Error opening database. If the binary or configuration has changed, replay the blockchain explicitly using `--force-replay`.
2022-10-24T13:02:11.559 chain_plugin.cpp:670 open ] If you know what you are doing you can skip this check and force open the database using `--force-open`.
2022-10-24T13:02:11.559 chain_plugin.cpp:671 open ] WARNING: THIS MAY CORRUPT YOUR DATABASE. FORCE OPEN AT YOUR OWN RISK.
2022-10-24T13:02:11.559 chain_plugin.cpp:672 open ] Error: {"code":10,"name":"assert_exception","message":"Assert Exception","stack":[{"context":{"level":"error","file":"database.cpp","line":198,"method":"load_state_initial_data","hostname":"","timestamp":"2022-10-24T13:02:11"},"format":"hb >= last_irreversible_block: ","data":{}},{"context":{"level":"warn","file":"database.cpp","line":155,"method":"open","hostname":"","timestamp":"2022-10-24T13:02:11"},"format":"rethrow","data":{"args.data_dir":"/home/hive/.hived/blockchain","args.shared_mem_dir":"/run/hive","args.shared_file_size":"25769803776"}}]}