included in already merged: !34 (merged)
In file included from /home/hive/psql_tools/tests/unit/mockups/mock/postgres_mock.cpp:1:0:
/home/hive/psql_tools/tests/unit/mockups/mock/postgres_mock.hpp:8:10: fatal error: gmock/gmock.h: No such file or directory
#include <gmock/gmock.h>
^~~~~~~~~~~~~~~
There was an experiment on steem-4 which dumping 56M of blocks to hive_fork_manager. At end of sync function 'hive.enable_indexes_of_irreversible()' last 11h. The function creates indexes and FKs for irreversible blocks tables. It looks like that tables 'hive.transactions' and 'hive.operations' are responsible for the problem.
Size of the whole DB: 1438GB
A table hive.operations contains (for 56M blocks) 2'733'086'127 rows and two indexes:
A table hive.transactions contains (for 56M blocks) 1'538'787'405 rows and two indexes:
In the HFM documentation there must be a place for some advice/guidelines about writing the application, examples of advice:
I would be good to test SQL cod with a linter
We can dump the hive_fork_manager database, and the use it to restore the database. But at the moment it is impossible to restore the whole HAF, because the state of the dumped db must correspond to the state of the hived ( hived must be restored to the same point where it was just before the db dump)
Before adding shared_lib to hive_fork_manager, target extension.hive_fork_manager did not compile C++ code and it was reasonable to compile unit test and mocks (C++) in the separated build step, but now hive_fork_manager depends on protocol lib which compilation lasts a while..., so there must be one step which compile sources before start tests
Unfortunately this is not easily reproducible, ie. fail appears 3 times after circa 100 runs
We can see this in database
psql -p 35432 -d haf_block_log_fc344f67adb743f78cd7ff3155cf53ef -h hive-3 -U myuser
password: mypassword
id | block_num | trx_in_block | op_pos | op_type_id | timestamp | body
-----+-----------+--------------+--------+------------+---------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
227 | 101 | -1 | 0 | 64 | 2021-10-15 10:53:00 | {"type":"producer_reward_operation","value":{"producer":"witness11-alpha","vesting_shares":{"amount":"7817","precision":6,"nai":"@@000000037"}}}
236 | 102 | -1 | 0 | 64 | 2021-10-15 10:53:06 | {"type":"producer_reward_operation","value":{"producer":"witness7-alpha","vesting_shares":{"amount":"7817","precision":6,"nai":"@@000000037"}}}
237 | 103 | -1 | 0 | 64 | 2021-10-15 10:53:09 | {"type":"producer_reward_operation","value":{"producer":"witness2-alpha","vesting_shares":{"amount":"7817","precision":6,"nai":"@@000000037"}}}
238 | 104 | -1 | 0 | 64 | 2021-10-15 10:53:12 | {"type":"producer_reward_operation","value":{"producer":"witness0-alpha","vesting_shares":{"amount":"7817","precision":6,"nai":"@@000000037"}}}
239 | 105 | -1 | 0 | 64 | 2021-10-15 10:53:15 | {"type":"producer_reward_operation","value":{"producer":"witness1-alpha","vesting_shares":{"amount":"7817","precision":6,"nai":"@@000000037"}}}
240 | 106 | -1 | 0 | 64 | 2021-10-15 10:53:21 | {"type":"producer_reward_operation","value":{"producer":"witness5-alpha","vesting_shares":{"amount":"7816","precision":6,"nai":"@@000000037"}}}
241 | 107 | -1 | 0 | 64 | 2021-10-15 10:53:24 | {"type":"producer_reward_operation","value":{"producer":"witness7-alpha","vesting_shares":{"amount":"7816","precision":6,"nai":"@@000000037"}}}
243 | 108 | -1 | 0 | 3 | 2021-10-15 10:53:27 | {"type":"transfer_to_vesting_operation","value":{"from":"initminer","to":"","amount":{"amount":"1234000","precision":3,"nai":"@@000000021"}}}
242 | 108 | -1 | 0 | 64 | 2021-10-15 10:53:27 | {"type":"producer_reward_operation","value":{"producer":"witness9-alpha","vesting_shares":{"amount":"39163","precision":6,"nai":"@@000000037"}}}
244 | 108 | -1 | 0 | 77 | 2021-10-15 10:53:27 | {"type":"transfer_to_vesting_completed_operation","value":{"from_account":"initminer","to_account":"initminer","hive_vested":{"amount":"1234000","precision":3,"nai":"@@000000021"},"vesting_shares_received":{"amount":"49213172","precision":6,"nai":"@@000000037"}}}
245 | 109 | -1 | 1 | 64 | 2021-10-15 10:53:30 | {"type":"producer_reward_operation","value":{"producer":"witness10-alpha","vesting_shares":{"amount":"7816","precision":6,"nai":"@@000000037"}}}
246 | 110 | -1 | 0 | 64 | 2021-10-15 10:53:42 | {"type":"producer_reward_operation","value":{"producer":"witness2-alpha","vesting_shares":{"amount":"7816","precision":6,"nai":"@@000000037"}}}
247 | 111 | -1 | 0 | 64 | 2021-10-15 10:53:48 | {"type":"producer_reward_operation","value":{"producer":"witness6-alpha","vesting_shares":{"amount":"7816","precision":6,"nai":"@@000000037"}}}
248 | 112 | -1 | 0 | 64 | 2021-10-15 10:53:51 | {"type":"producer_reward_operation","value":{"producer":"witness4-beta","vesting_shares":{"amount":"7816","precision":6,"nai":"@@000000037"}}}
(14 rows)
block_num | trx_in_block | trx_hash | ref_block_num | ref_block_prefix | expiration | signature
-----------+--------------+------------------------------------------------------------------------------------+---------------+------------------+---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
44 | 0 | \x38353665616436666332653662633764326536303065663430633664623433393434373836323634 | 43 | 1303795060 | 2021-10-15 10:50:36 | \x31663231646231383830323838653233376133346332383231343665663431643032613430383361373532616232636536343938363464396332353363653663313336353231383339656563336261636361323839623362653265623863303765623239633562623963386237656363393866373532343064303966633131633133
45 | 0 | \x66313039353737316538323163633236613532653238663764346536386235653931346166353064 | 44 | 1207641821 | 2021-10-15 10:50:39 | \x32303765656134636235316638623863393364383330343530643437636232336636366430653030633335376465313039653030383765616431363861306536346131336237396266316366346431656162663264396464616434376633383831373265343866326463373562376564366530313931663734626138653338633533
46 | 0 | \x64346161643131313237306430353833333466383865323163663266373431356430303462653437 | 45 | 3958641219 | 2021-10-15 10:50:42 | \x31663162633230306434386464333632353463653730643661393733643035303537613734363566376261666366613166356537313735626331373834633766363030323866303264353033386430326265643465356333363631363264333666616633356565323334663733633735383063356562303233393232356534396233
(3 rows)
There is transfer_to_vesting_operation in hive.operations and its operation from different fork, and this should be reverted. We can see also that there is no corresponding transaction in hive.transactions - it means that this transaction was succesfully reverted.
There is a need to make some method in the hived API which initializes some internals of hive fork manager when a new session with hived is starting ( first connect to the HFM, or connect after rebooting the systems, crash etc... ). At the moment I see two things to do in the function:
Even more useful, pg_dump knows that it should not dump the individual member objects of the extension — it will just include a CREATE EXTENSION command in dumps, instead. This vastly simplifies migration to a new version of the extension that might contain more or different objects than the old version.
https://www.postgresql.org/docs/12/extend-extensions.html
Ordinarily, if a table is part of an extension, neither the table's definition nor its content will be dumped by pg_dump.
The hived algorithm must be extended for the session initialization:
Update: run test 1000 times using newest master and postgres 12 (previously it was postgres 10) and it didn't happen again, to be sure I leave it for one more night
Marcin Sobczyk (f51161da) at 22 Oct 09:34
temporary commit for issue 27
Unfortunately this is not easily reproducible, ie. fail appears 3 times after circa 100 runs
We can see this in database
psql -p 35432 -d haf_block_log_fc344f67adb743f78cd7ff3155cf53ef -h hive-3 -U myuser
password: mypassword
id | block_num | trx_in_block | op_pos | op_type_id | timestamp | body
-----+-----------+--------------+--------+------------+---------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
227 | 101 | -1 | 0 | 64 | 2021-10-15 10:53:00 | {"type":"producer_reward_operation","value":{"producer":"witness11-alpha","vesting_shares":{"amount":"7817","precision":6,"nai":"@@000000037"}}}
236 | 102 | -1 | 0 | 64 | 2021-10-15 10:53:06 | {"type":"producer_reward_operation","value":{"producer":"witness7-alpha","vesting_shares":{"amount":"7817","precision":6,"nai":"@@000000037"}}}
237 | 103 | -1 | 0 | 64 | 2021-10-15 10:53:09 | {"type":"producer_reward_operation","value":{"producer":"witness2-alpha","vesting_shares":{"amount":"7817","precision":6,"nai":"@@000000037"}}}
238 | 104 | -1 | 0 | 64 | 2021-10-15 10:53:12 | {"type":"producer_reward_operation","value":{"producer":"witness0-alpha","vesting_shares":{"amount":"7817","precision":6,"nai":"@@000000037"}}}
239 | 105 | -1 | 0 | 64 | 2021-10-15 10:53:15 | {"type":"producer_reward_operation","value":{"producer":"witness1-alpha","vesting_shares":{"amount":"7817","precision":6,"nai":"@@000000037"}}}
240 | 106 | -1 | 0 | 64 | 2021-10-15 10:53:21 | {"type":"producer_reward_operation","value":{"producer":"witness5-alpha","vesting_shares":{"amount":"7816","precision":6,"nai":"@@000000037"}}}
241 | 107 | -1 | 0 | 64 | 2021-10-15 10:53:24 | {"type":"producer_reward_operation","value":{"producer":"witness7-alpha","vesting_shares":{"amount":"7816","precision":6,"nai":"@@000000037"}}}
243 | 108 | -1 | 0 | 3 | 2021-10-15 10:53:27 | {"type":"transfer_to_vesting_operation","value":{"from":"initminer","to":"","amount":{"amount":"1234000","precision":3,"nai":"@@000000021"}}}
242 | 108 | -1 | 0 | 64 | 2021-10-15 10:53:27 | {"type":"producer_reward_operation","value":{"producer":"witness9-alpha","vesting_shares":{"amount":"39163","precision":6,"nai":"@@000000037"}}}
244 | 108 | -1 | 0 | 77 | 2021-10-15 10:53:27 | {"type":"transfer_to_vesting_completed_operation","value":{"from_account":"initminer","to_account":"initminer","hive_vested":{"amount":"1234000","precision":3,"nai":"@@000000021"},"vesting_shares_received":{"amount":"49213172","precision":6,"nai":"@@000000037"}}}
245 | 109 | -1 | 1 | 64 | 2021-10-15 10:53:30 | {"type":"producer_reward_operation","value":{"producer":"witness10-alpha","vesting_shares":{"amount":"7816","precision":6,"nai":"@@000000037"}}}
246 | 110 | -1 | 0 | 64 | 2021-10-15 10:53:42 | {"type":"producer_reward_operation","value":{"producer":"witness2-alpha","vesting_shares":{"amount":"7816","precision":6,"nai":"@@000000037"}}}
247 | 111 | -1 | 0 | 64 | 2021-10-15 10:53:48 | {"type":"producer_reward_operation","value":{"producer":"witness6-alpha","vesting_shares":{"amount":"7816","precision":6,"nai":"@@000000037"}}}
248 | 112 | -1 | 0 | 64 | 2021-10-15 10:53:51 | {"type":"producer_reward_operation","value":{"producer":"witness4-beta","vesting_shares":{"amount":"7816","precision":6,"nai":"@@000000037"}}}
(14 rows)
block_num | trx_in_block | trx_hash | ref_block_num | ref_block_prefix | expiration | signature
-----------+--------------+------------------------------------------------------------------------------------+---------------+------------------+---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
44 | 0 | \x38353665616436666332653662633764326536303065663430633664623433393434373836323634 | 43 | 1303795060 | 2021-10-15 10:50:36 | \x31663231646231383830323838653233376133346332383231343665663431643032613430383361373532616232636536343938363464396332353363653663313336353231383339656563336261636361323839623362653265623863303765623239633562623963386237656363393866373532343064303966633131633133
45 | 0 | \x66313039353737316538323163633236613532653238663764346536386235653931346166353064 | 44 | 1207641821 | 2021-10-15 10:50:39 | \x32303765656134636235316638623863393364383330343530643437636232336636366430653030633335376465313039653030383765616431363861306536346131336237396266316366346431656162663264396464616434376633383831373265343866326463373562376564366530313931663734626138653338633533
46 | 0 | \x64346161643131313237306430353833333466383865323163663266373431356430303462653437 | 45 | 3958641219 | 2021-10-15 10:50:42 | \x31663162633230306434386464333632353463653730643661393733643035303537613734363566376261666366613166356537313735626331373834633766363030323866303264353033386430326265643465356333363631363264333666616633356565323334663733633735383063356562303233393232356534396233
(3 rows)
There is transfer_to_vesting_operation in hive.operations and its operation from different fork, and this should be reverted. We can see also that there is no corresponding transaction in hive.transactions - it means that this transaction was succesfully reverted.
Marcin (7ac0d824) at 21 Oct 09:32
Marcin (4aebed17) at 21 Oct 09:32
Merge branch 'bw_use_pkgconfig_instead_of_findpackage' into 'master'
... and 3 more commits
Marcin (7ac0d824) at 21 Oct 09:25
cmake fixes: temporary remove static build of libs, support for PQ
... and 8 more commits
Marcin (08a6f99e) at 21 Oct 09:17
cmake fixes: temporary remove static build of libs, support for PQ
Mariusz Trela (c76cbf5d) at 21 Oct 08:51