HAF recovery after loss of ram-based statefile
A haf server (shed14) with a fully synced hivemind and ramdisk-based state file lost power (bad UPS). Docker images: shed14: 12/21 replayed thread-names with older hive ON hmind_howo 631b0543
HAF_IMAGE=registry.gitlab.syncad.com/hive/haf/instance:6671c63
Since the database was ok, I just did a replay-blockchain
of hived to fix the statefile. After replay, hived died while trying to restore indexes:
2023-12-28T07:46:10.914081 indexes_controler.cpp:117 operator() ] Attempting to execute query: `SELECT hive.restore_indexes( 'hive.applied_hardforks' );`...
2023-12-28T07:46:10.914158 indexes_controler.cpp:120 operator() ] Query processor: `SELECT hive.restore_indexes( 'hive.irreversible_data' );' Creating of enable indexes done in 1.52499999999999991 ms
2023-12-28T07:46:10.914177 indexes_controler.cpp:124 operator() ] The enable indexes have been created...
2023-12-28T07:46:10.942131 data_processor.cpp:214 handle_exception ] Data processor Query processor: `SELECT hive.restore_indexes( 'hive.applied_hardforks' );' detected SQL statement execution failure. Failing statement: `SELECT hive.restore_indexes( 'hive.applied_hardforks' );'.
2023-12-28T07:46:10.942180 data_processor.cpp:16 kill_node ] An error occured and HAF is stopping synchronization...
Maybe some ordering problem related to how constraints are restored?
haf_block_log=# \dS+ hive.blocks
Table "hive.blocks"
Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
-------------------------+-----------------------------+-----------+----------+---------+----------+-------------+--------------+-------------
num | integer | | not null | | plain | | |
hash | bytea | | not null | | extended | | |
prev | bytea | | not null | | extended | | |
created_at | timestamp without time zone | | not null | | plain | | |
producer_account_id | integer | | not null | | plain | | |
transaction_merkle_root | bytea | | not null | | extended | | |
extensions | jsonb | | | | extended | | |
witness_signature | bytea | | not null | | extended | | |
signing_key | text | | not null | | extended | | |
hbd_interest_rate | hive.interest_rate | | | | plain | | |
total_vesting_fund_hive | hive.hive_amount | | | | main | | |
total_vesting_shares | hive.vest_amount | | | | main | | |
total_reward_fund_hive | hive.hive_amount | | | | main | | |
virtual_supply | hive.hive_amount | | | | main | | |
current_supply | hive.hive_amount | | | | main | | |
current_hbd_supply | hive.hbd_amount | | | | main | | |
dhf_interval_ledger | hive.hbd_amount | | | | main | | |
Access method: heap
haf_block_log=# \dS+ hive.irreversible_data
Table "hive.irreversible_data"
Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
------------------+---------+-----------+----------+---------+---------+-------------+--------------+-------------
id | integer | | not null | | plain | | |
consistent_block | integer | | | | plain | | |
is_dirty | boolean | | not null | | plain | | |
Indexes:
"pk_irreversible_data" PRIMARY KEY, btree (id)
Access method: heap
haf_block_log=# ALTER TABLE hive.irreversible_data ADD CONSTRAINT fk_1_hive_irreversible_data FOREIGN KEY (consistent_block) REFERENCES hive.blocks(num) NOT VALID;
ERROR: there is no unique constraint matching given keys for referenced table "blocks"
table_name | index_constraint_name | command | is_constraint | is_index | is_foreign_key
----------------------------+----------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------+----------+----------------
hive.irreversible_data | fk_1_hive_irreversible_data | ALTER TABLE hive.irreversible_data ADD CONSTRAINT fk_1_hive_irreversible_data FOREIGN KEY (consistent_block) REFERENCES hive.blocks(num) NOT VALID | f | f | t
hive.blocks | fk_1_hive_blocks | ALTER TABLE hive.blocks ADD CONSTRAINT fk_1_hive_blocks FOREIGN KEY (producer_account_id) REFERENCES hive.accounts(id) DEFERRABLE INITIALLY DEFERRED NOT VALID | f | f | t
hive.transactions | fk_1_hive_transactions | ALTER TABLE hive.transactions ADD CONSTRAINT fk_1_hive_transactions FOREIGN KEY (block_num) REFERENCES hive.blocks(num) NOT VALID | f | f | t
hive.transactions_multisig | fk_1_hive_transactions_multisig | ALTER TABLE hive.transactions_multisig ADD CONSTRAINT fk_1_hive_transactions_multisig FOREIGN KEY (trx_hash) REFERENCES hive.transactions(trx_hash) NOT VALID | f | f | t
hive.operations | fk_1_hive_operations | ALTER TABLE hive.operations ADD CONSTRAINT fk_1_hive_operations FOREIGN KEY (block_num) REFERENCES hive.blocks(num) NOT VALID | f | f | t
hive.operations | fk_2_hive_operations | ALTER TABLE hive.operations ADD CONSTRAINT fk_2_hive_operations FOREIGN KEY (op_type_id) REFERENCES hive.operation_types(id) NOT VALID | f | f | t
hive.applied_hardforks | fk_1_hive_applied_hardforks | ALTER TABLE hive.applied_hardforks ADD CONSTRAINT fk_1_hive_applied_hardforks FOREIGN KEY (hardfork_vop_id) REFERENCES hive.operations(id) NOT VALID | f | f | t
hive.applied_hardforks | fk_2_hive_applied_hardforks | ALTER TABLE hive.applied_hardforks ADD CONSTRAINT fk_2_hive_applied_hardforks FOREIGN KEY (block_num) REFERENCES hive.blocks(num) NOT VALID | f | f | t
hive.accounts | fk_1_hive_accounts | ALTER TABLE hive.accounts ADD CONSTRAINT fk_1_hive_accounts FOREIGN KEY (block_num) REFERENCES hive.blocks(num) MATCH FULL NOT VALID | f | f | t
hive.account_operations | hive_account_operations_fk_1 | ALTER TABLE hive.account_operations ADD CONSTRAINT hive_account_operations_fk_1 FOREIGN KEY (account_id) REFERENCES hive.accounts(id) NOT VALID | f | f | t
hive.account_operations | hive_account_operations_fk_2 | ALTER TABLE hive.account_operations ADD CONSTRAINT hive_account_operations_fk_2 FOREIGN KEY (operation_id) REFERENCES hive.operations(id) NOT VALID | f | f | t
hive.account_operations | hive_account_operations_fk_3 | ALTER TABLE hive.account_operations ADD CONSTRAINT hive_account_operations_fk_3 FOREIGN KEY (op_type_id) REFERENCES hive.operation_types(id) NOT VALID | f | f | t
hive.blocks | pk_hive_blocks | ALTER TABLE hive.blocks ADD CONSTRAINT pk_hive_blocks PRIMARY KEY (num) | t | f | f
hive.blocks | hive_blocks_created_at_idx | CREATE INDEX hive_blocks_created_at_idx ON hive.blocks USING btree (created_at) | f | t | f
hive.blocks | hive_blocks_producer_account_id_idx | CREATE INDEX hive_blocks_producer_account_id_idx ON hive.blocks USING btree (producer_account_id) | f | t | f
hive.transactions | pk_hive_transactions | ALTER TABLE hive.transactions ADD CONSTRAINT pk_hive_transactions PRIMARY KEY (trx_hash) | t | f | f
hive.transactions | hive_transactions_block_num_trx_in_block_idx | CREATE INDEX hive_transactions_block_num_trx_in_block_idx ON hive.transactions USING btree (block_num, trx_in_block) | f | t | f
hive.transactions_multisig | pk_hive_transactions_multisig | ALTER TABLE hive.transactions_multisig ADD CONSTRAINT pk_hive_transactions_multisig PRIMARY KEY (trx_hash, signature) | t | f | f
hive.applied_hardforks | pk_hive_applied_hardforks | ALTER TABLE hive.applied_hardforks ADD CONSTRAINT pk_hive_applied_hardforks PRIMARY KEY (hardfork_num) | t | f | f
hive.applied_hardforks | hive_applied_hardforks_block_num_idx | CREATE INDEX hive_applied_hardforks_block_num_idx ON hive.applied_hardforks USING btree (block_num) | f | t | f
hive.accounts | pk_hive_accounts_id | ALTER TABLE hive.accounts ADD CONSTRAINT pk_hive_accounts_id PRIMARY KEY (id) | t | f | f
hive.accounts | uq_hive_accounst_name | ALTER TABLE hive.accounts ADD CONSTRAINT uq_hive_accounst_name UNIQUE (name) | t | f | f
hive.accounts | hive_accounts_block_num_idx | CREATE INDEX hive_accounts_block_num_idx ON hive.accounts USING btree (block_num) | f | t | f
hive.account_operations | hive_account_operations_uq2 | ALTER TABLE hive.account_operations ADD CONSTRAINT hive_account_operations_uq2 UNIQUE (account_id, operation_id) | t | f | f
hive.blocks | fk_1_hive_blocks | ALTER TABLE hive.blocks ADD CONSTRAINT fk_1_hive_blocks FOREIGN KEY (producer_account_id) REFERENCES hive.accounts(id) DEFERRABLE INITIALLY DEFERRED NOT VALID | f | f | t
hive.transactions | fk_1_hive_transactions | ALTER TABLE hive.transactions ADD CONSTRAINT fk_1_hive_transactions FOREIGN KEY (block_num) REFERENCES hive.blocks(num) NOT VALID | f | f | t
hive.transactions_multisig | fk_1_hive_transactions_multisig | ALTER TABLE hive.transactions_multisig ADD CONSTRAINT fk_1_hive_transactions_multisig FOREIGN KEY (trx_hash) REFERENCES hive.transactions(trx_hash) NOT VALID | f | f | t
hive.operations | fk_1_hive_operations | ALTER TABLE hive.operations ADD CONSTRAINT fk_1_hive_operations FOREIGN KEY (block_num) REFERENCES hive.blocks(num) NOT VALID | f | f | t
hive.operations | fk_2_hive_operations | ALTER TABLE hive.operations ADD CONSTRAINT fk_2_hive_operations FOREIGN KEY (op_type_id) REFERENCES hive.operation_types(id) NOT VALID | f | f | t
hive.applied_hardforks | fk_1_hive_applied_hardforks | ALTER TABLE hive.applied_hardforks ADD CONSTRAINT fk_1_hive_applied_hardforks FOREIGN KEY (hardfork_vop_id) REFERENCES hive.operations(id) NOT VALID | f | f | t
hive.applied_hardforks | fk_2_hive_applied_hardforks | ALTER TABLE hive.applied_hardforks ADD CONSTRAINT fk_2_hive_applied_hardforks FOREIGN KEY (block_num) REFERENCES hive.blocks(num) NOT VALID | f | f | t
hive.accounts | fk_1_hive_accounts | ALTER TABLE hive.accounts ADD CONSTRAINT fk_1_hive_accounts FOREIGN KEY (block_num) REFERENCES hive.blocks(num) MATCH FULL NOT VALID | f | f | t
hive.account_operations | hive_account_operations_fk_1 | ALTER TABLE hive.account_operations ADD CONSTRAINT hive_account_operations_fk_1 FOREIGN KEY (account_id) REFERENCES hive.accounts(id) NOT VALID | f | f | t
hive.account_operations | hive_account_operations_fk_2 | ALTER TABLE hive.account_operations ADD CONSTRAINT hive_account_operations_fk_2 FOREIGN KEY (operation_id) REFERENCES hive.operations(id) NOT VALID | f | f | t
hive.account_operations | hive_account_operations_fk_3 | ALTER TABLE hive.account_operations ADD CONSTRAINT hive_account_operations_fk_3 FOREIGN KEY (op_type_id) REFERENCES hive.operation_types(id) NOT VALID | f | f | t
hive.blocks | pk_hive_blocks | ALTER TABLE hive.blocks ADD CONSTRAINT pk_hive_blocks PRIMARY KEY (num) | t | f | f
hive.blocks | hive_blocks_created_at_idx | CREATE INDEX hive_blocks_created_at_idx ON hive.blocks USING btree (created_at) | f | t | f
hive.blocks | hive_blocks_producer_account_id_idx | CREATE INDEX hive_blocks_producer_account_id_idx ON hive.blocks USING btree (producer_account_id) | f | t | f
hive.transactions | pk_hive_transactions | ALTER TABLE hive.transactions ADD CONSTRAINT pk_hive_transactions PRIMARY KEY (trx_hash) | t | f | f
hive.transactions | hive_transactions_block_num_trx_in_block_idx | CREATE INDEX hive_transactions_block_num_trx_in_block_idx ON hive.transactions USING btree (block_num, trx_in_block) | f | t | f
hive.transactions_multisig | pk_hive_transactions_multisig | ALTER TABLE hive.transactions_multisig ADD CONSTRAINT pk_hive_transactions_multisig PRIMARY KEY (trx_hash, signature) | t | f | f
hive.applied_hardforks | pk_hive_applied_hardforks | ALTER TABLE hive.applied_hardforks ADD CONSTRAINT pk_hive_applied_hardforks PRIMARY KEY (hardfork_num) | t | f | f
hive.applied_hardforks | hive_applied_hardforks_block_num_idx | CREATE INDEX hive_applied_hardforks_block_num_idx ON hive.applied_hardforks USING btree (block_num) | f | t | f
hive.accounts | pk_hive_accounts_id | ALTER TABLE hive.accounts ADD CONSTRAINT pk_hive_accounts_id PRIMARY KEY (id) | t | f | f
hive.accounts | uq_hive_accounst_name | ALTER TABLE hive.accounts ADD CONSTRAINT uq_hive_accounst_name UNIQUE (name) | t | f | f
hive.accounts | hive_accounts_block_num_idx | CREATE INDEX hive_accounts_block_num_idx ON hive.accounts USING btree (block_num) | f | t | f
hive.account_operations | hive_account_operations_uq2 | ALTER TABLE hive.account_operations ADD CONSTRAINT hive_account_operations_uq2 UNIQUE (account_id, operation_id) | t | f | f
hive.account_operations | hive_account_operations_uq_1 | ALTER TABLE hive.account_operations ADD CONSTRAINT hive_account_operations_uq_1 UNIQUE (account_id, account_op_seq_no) | t | f | f
hive.account_operations | hive_account_operations_type_account_id_op_seq_idx | CREATE UNIQUE INDEX hive_account_operations_type_account_id_op_seq_idx ON hive.account_operations USING btree (op_type_id, account_id, account_op_seq_no DESC) INCLUDE (operation_id, block_num) | f | t | f
(26 rows)
Edited by Dan Notestein