Skip to content
Snippets Groups Projects
New look: Off

hivemind replay crash on s13: probably a couple restarts on this server, maybe base haf was corrupted (but if so, we probably could use some internal consistency test to determine such cases)

    • Closed Issue created by Dan Notestein @dan 
    INFO - hive.indexer.sync:383 - [MASSIVE] Got block 18989000 @ 2018-01-15T03:57:21 (84/s, 84rps, 540111581wps) -- done in 11s, avg rate: 84.2/s
INFO - hive.indexer.sync:384 - [MASSIVE] Time elapsed: 28509.935351923108s
INFO - hive.indexer.sync:385 - [MASSIVE] Current system time: 2024-11-12 14:05:53.206
INFO - hive.indexer.sync:386 - memory usage report: physical_memory = 640.00 MB, virtual_memory = 1266.50 MB, shared_memory = 3.34 MB
INFO - hive.utils.stats:162 - ####################
INFO - hive.utils.stats:213 - Operations present in the processed blocks
INFO - hive.utils.stats:156 - `OperationType.COMMENT`: Processed 7459 times in 7.54979 seconds
INFO - hive.utils.stats:156 - `OperationType.CUSTOM_JSON`: Processed 14175 times in 0.56902 seconds
INFO - hive.utils.stats:156 - `VirtualOperationType.EFFECTIVE_COMMENT_VOTE`: Processed 18542 times in 0.39947 seconds
INFO - hive.utils.stats:156 - `OperationType.COMMENT_OPTION`: Processed 668 times in 0.34914 seconds
INFO - hive.utils.stats:156 - `OperationType.VOTE`: Processed 23842 times in 0.28625 seconds
INFO - hive.utils.stats:156 - `VirtualOperationType.COMMENT_PAYOUT_UPDATE`: Processed 6721 times in 0.04436 seconds
INFO - hive.utils.stats:156 - `OperationType.DELETE_COMMENT`: Processed 73 times in 0.03925 seconds
INFO - hive.utils.stats:156 - `OperationType.TRANSFER`: Processed 1112 times in 0.02007 seconds
INFO - hive.utils.stats:156 - `VirtualOperationType.AUTHOR_REWARD`: Processed 1684 times in 0.01824 seconds
INFO - hive.utils.stats:156 - `VirtualOperationType.COMMENT_REWARD`: Processed 1684 times in 0.01579 seconds
INFO - hive.utils.stats:156 - `OperationType.ACCOUNT_CREATE_WITH_DELEGATION`: Processed 33 times in 0.01493 seconds
INFO - hive.utils.stats:156 - `OperationType.ACCOUNT_UPDATE`: Processed 233 times in 0.00294 seconds
INFO - hive.utils.stats:156 - `OperationType.ACCOUNT_CREATE`: Processed 1 times in 0.00041 seconds
INFO - hive.utils.stats:215 - Current time for processing operations time: 9.3097s.
INFO - hive.utils.stats:162 - ####################
INFO - hive.utils.stats:265 - Flushing times
INFO - hive.utils.stats:156 - `PostDataCache`: Pushed 6823 records in 2.0989 seconds
INFO - hive.utils.stats:156 - `Votes`: Pushed 23011 records in 1.7104 seconds
INFO - hive.utils.stats:156 - `Posts`: Pushed 13442 records in 1.0941 seconds
INFO - hive.utils.stats:156 - `Follow`: Pushed 13334 records in 0.8609 seconds
INFO - hive.utils.stats:156 - `Reblog`: Pushed 508 records in 0.1418 seconds
INFO - hive.utils.stats:156 - `Accounts`: Pushed 152 records in 0.0597 seconds
INFO - hive.utils.stats:156 - `Notify`: Pushed 0 records in 0.0000 seconds
INFO - hive.utils.stats:267 - Current flushing time: 5.9658s.
INFO - hive.utils.stats:162 - ####################
INFO - hive.utils.stats:350 - Waiting times
INFO - hive.utils.stats:156 - `block_consumer_block`: Waited 1.4970 seconds
INFO - hive.utils.stats:352 - Current waiting time: 1.4970s.
INFO - hive.indexer.sync:393 - Calculated time: 16.7724 s.
INFO - hive.indexer.sync:122 - Last imported block is: 18990000
INFO - hive.indexer.blocks:145 - End-of-sync LIB is set to 90622517, last block that guarantees cashout at end of sync is 90420917
INFO - hive.indexer.sync:214 - Querying for next block for app context...
INFO - hive.indexer.sync:238 - Next block range from hive.app_next_iteration is: <18990001:18991000>
INFO - hive.indexer.sync:138 - target_head_block: 18991000
INFO - hive.indexer.sync:139 - test_max_block: None
INFO - hive.indexer.follow:277 - follow_op {'follower': 'lauralvarado', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:260 - follow_op {'follower': 'topkpop', 'following': 'reinhard-schmid', 'what': ['blog']} ignored due to invalid follower
INFO - hive.indexer.follow:277 - follow_op {'follower': 'ammadkhalid', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
INFO - hive.indexer.follow:277 - follow_op {'follower': 'viki.beta', 'following': [], 'what': ['blog']} is void due to effectively empty list of following
ERROR - hive.indexer.blocks:202 - exception encountered block 18989635
ERROR - hive.indexer.sync:404 - Exception caught during processing blocks...
Traceback (most recent call last):
  File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/hive/indexer/sync.py", line 371, in _consume_massive_blocks
    Blocks.process_multi(blocks, True)
  File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/hive/indexer/blocks.py", line 219, in process_multi
    first_block, last_num = cls.process_blocks(blocks)
  File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/hive/indexer/blocks.py", line 203, in process_blocks
    raise e
  File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/hive/indexer/blocks.py", line 200, in process_blocks
    last_num = cls._process(hiveBlock)
  File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/hive/indexer/blocks.py", line 417, in _process
    Payments.op_transfer(op, transaction.get_id(), num, cls._head_block_date)
  File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/hive/indexer/payments.py", line 20, in op_transfer
    result = cls._validated(op, tx_idx, num, date)
  File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/hive/indexer/payments.py", line 90, in _validated
    'from_account': Accounts.get_id(op['from']),
  File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/hive/indexer/accounts.py", line 86, in get_id
    assert name in cls._ids, f'Account \'{name}\' does not exist'
AssertionError: Account 'coreycck' does not exist
INFO - hive.indexer.sync:85 - Exiting HAF mode synchronization
WARNING - hive.server.common.payout_stats:16 - Rebuilding payout_stats_view in separate transaction
INFO - hive.indexer.sync:90 - LAST IMPORTED BLOCK IS: 18991000
INFO - hive.indexer.sync:91 - LAST COMPLETED BLOCK IS: 1
INFO - hive.db.adapter:98 - Closing database connection: 'massive_instance'
INFO - hive.db.adapter:98 - Closing database connection: 'PostDataCache'
INFO - hive.db.adapter:98 - Closing database connection: 'Votes'
INFO - hive.db.adapter:98 - Closing database connection: 'Follow'
INFO - hive.db.adapter:98 - Closing database connection: 'Posts'
INFO - hive.db.adapter:98 - Closing database connection: 'Reblog'
INFO - hive.db.adapter:98 - Closing database connection: 'Notify'
INFO - hive.db.adapter:98 - Closing database connection: 'Accounts'
INFO - hive.db.adapter:98 - Closing database connection: 'Mentions'
INFO - hive.db.adapter:98 - Closing database connection: 'root'
INFO - hive.db.adapter:109 - Disposing SQL engine
INFO - hive.conf:274 - The database is disconnected...
Traceback (most recent call last):
  File "/home/hivemind/.hivemind-venv/bin/hive", line 8, in <module>
    sys.exit(run())
  File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/hive/cli.py", line 65, in run
    launch_mode(mode, conf)
  File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/hive/cli.py", line 107, in launch_mode
    sync.run()
  File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/hive/indexer/sync.py", line 155, in run
    self._process_massive_blocks(self._lbound, self._ubound, active_connections_before)
  File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/hive/indexer/sync.py", line 263, in _process_massive_blocks
    self._wait_for_massive_consume() # wait for finish previous consumption
  File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/hive/indexer/sync.py", line 192, in _wait_for_massive_consume
    self._massive_consume_blocks_futures.result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 444, in result
    return self.__get_result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/hive/indexer/sync.py", line 371, in _consume_massive_blocks
    Blocks.process_multi(blocks, True)
  File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/hive/indexer/blocks.py", line 219, in process_multi
    first_block, last_num = cls.process_blocks(blocks)
  File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/hive/indexer/blocks.py", line 203, in process_blocks
    raise e
  File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/hive/indexer/blocks.py", line 200, in process_blocks
    last_num = cls._process(hiveBlock)
  File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/hive/indexer/blocks.py", line 417, in _process
    Payments.op_transfer(op, transaction.get_id(), num, cls._head_block_date)
  File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/hive/indexer/payments.py", line 20, in op_transfer
    result = cls._validated(op, tx_idx, num, date)
  File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/hive/indexer/payments.py", line 90, in _validated
    'from_account': Accounts.get_id(op['from']),
  File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/hive/indexer/accounts.py", line 86, in get_id
    assert name in cls._ids, f'Account \'{name}\' does not exist'
AssertionError: Account 'coreycck' does not exist

    Designs

      Child items ...

      2. Activity

      • Dan Notestein changed title from hivemind replay crash on s13: probably a couple restarts on this server, maybe base haf was corrupted (but if so, we probably could use some test to determine such cases) to hivemind replay crash on s13: probably a couple restarts on this server, maybe base haf was corrupted (but if so, we probably could use some internal consistency test to determine such cases)

        changed title from hivemind replay crash on s13: probably a couple restarts on this server, maybe base haf was corrupted (but if so, we probably could use some test to determine such cases) to hivemind replay crash on s13: probably a couple restarts on this server, maybe base haf was corrupted (but if so, we probably could use some internal consistency test to determine such cases)

      • Dan Notestein
        Dan Notestein @dan ·
        Author Owner

        Seems like 3 possibilities:

        • haf is corrupted due to failure to have a wal directory available (we need to add a check at startup for this in haf). This is a likely suspect since it is unique to this test run. But this should only likely cause a problem near the end of the blockchain and this problem occurs in the first 18m blocks.
        • hivemind is corrupted due to a restart. This hivemind run was interrupted/resumed several times while in massive sync.
        • hivemind has some race (this seems least likely)

        Ideas for diagnosis:

        • check if any blocks are missing from haf due to wal directory problem. This might be a good internal check to have.
        • If there's nothing obvious wrong with haf, we could uninstall hivemind and replay to see if we get the same error again. If the problem is due to hivemind being interrupted (currently my best guess), then it should replay ok after uninstall/reinstall.
        Edited by Dan Notestein
      • Bartek Wrona
        Bartek Wrona @wrona ·
        Owner

        coreycck account has been created in block 13974356: https://explore.openhive.network/transaction/557e949b816cb7665cffd9ff99778c97bee3efc1 quite far from failure block Question is at which blocks hivemind has been restarted earlier. Latest account operations (hivemind specific) before failure have been performed in block 16018772: https://explore.openhive.network/transaction/8738796903611ccaab0e9158cc70849bb29390f6

      • Bartek Wrona
        Bartek Wrona @wrona ·
        Owner

        Given account exists in HAF hafd.accounts table. But it is missing in hivemind_app.hive_accounts. For the other side it exists in tables specific to reputation_tracker and balance_tracker applications. So it rather points for some hivemind specific problem.

      • Marcin
        Marcin @Ickiewicz ·
        Developer

        SELECT * FROM hivemind_app.hive_votes WHERE block_num BETWEEN 13974001 AND 13975000; returns NULL, which means that there is a lack of data for 1000 of blocks in hivemind ( the data is in HAF tables). 1000 blocks is the size of the batch of blocks to process during massive sync. I suppose during breaking hivemind with SIGINT the changes in the context were committed but modifications in hivemind tables were not (they are filled in separate threads)

        Edited by Marcin
      • Dan Notestein
        Dan Notestein @dan ·
        Author Owner

        We've made what we hope was a fix for this, so closing for now.

      • Dan Notestein closed

        closed

      Please register or sign in to reply