Hivemind sync stalled

I have built HAF and Hivemind docker images from source. I run the HAF image and then the Hivemind image, in parallel. HAF is filling its db with a good pace, whereas Hivemind started very slowly and became even slower and has ground to a halt at just block 28,000.

In addition, if I want to start the Hivemind API server, I get a hive not found error.

Any help would be appreciated.

Here are the Hivemind logs:

/home/hivemind/app /home/hivemind
[1]+     9 Running                 { echo "Attempting to start Hivemind process..."; sudo -HEnu hivemind /bin/bash <<EOF
  source /home/hivemind/.hivemind-venv/bin/activate
  hive "${HIVEMIND_ARGS[@]}" --database-url="${POSTGRES_URL}"
EOF
 echo "Hivemind process finished execution: $?"; } &
waiting for job finish: 9.
Attempting to start Hivemind process...
INFO - hive.conf:205 - The database instance is created...
INFO - hive.db.adapter:48 - A database offers maximum connections: 100. Required 15 connections.
INFO - hive.db.adapter:90 - Closing database connection: 'root'
INFO - hive.db.adapter:103 - Disposing SQL engine
INFO - hive.conf:254 - The database is disconnected...
INFO - hive.conf:205 - The database instance is created...
INFO - hive.indexer.sync:52 - Entering HAF mode synchronization
INFO - hive.db.db_state:44 - [MASSIVE] Welcome to hive!
INFO - hive.db.db_state:55 - [MASSIVE] Continue with massive sync...
INFO - hive.indexer.sync:67 - hivemind_version : 1.27.3.0.0+g3bfe4820-dirty.20221220T062630
INFO - hive.indexer.sync:68 - hivemind_git_rev : 3bfe4820fbc726c589d314eca700147f834cc63f
INFO - hive.indexer.sync:69 - hivemind_git_date : 2022-12-20 06:26:30
INFO - hive.indexer.sync:71 - database_schema_version : 34
INFO - hive.indexer.sync:72 - database_patch_date : 2023-03-17 22:35:26.790629
INFO - hive.indexer.sync:73 - database_patched_to_revision : 9d2cc15bea71a39139abdf49569e0eac6dd0b970
INFO - hive.indexer.sync:75 - last_block_from_view : 28000
INFO - hive.indexer.sync:76 - last_imported_block : 28000
INFO - hive.indexer.sync:77 - last_completed_block : 1
INFO - hive.indexer.hive_db.haf_functions:34 - Context already attached - attaching skipped.
INFO - hive.indexer.sync:89 - Using HAF database as block data provider, pointed by url: 'postgresql://haf_app_admin@172.17.0.2:5432/haf_block_log'
INFO - hive.indexer.sync:99 - Last imported block is: 28000
INFO - hive.indexer.sync:186 - Querying for next block for app context...
INFO - hive.indexer.sync:188 - Next block range from hive.app_next_block is: <28001:7406908>
INFO - hive.indexer.sync:117 - target_head_block: 7406908
INFO - hive.indexer.sync:118 - test_max_block: None
INFO - hive.indexer.sync:122 - [MASSIVE] *** MASSIVE blocks processing ***
INFO - hive.db.db_state:202 - [MASSIVE] Begin pre-massive sync hooks for table hive_reputation_data
INFO - hive.db.db_state:202 - [MASSIVE] Begin pre-massive sync hooks for table hive_posts
INFO - hive.db.db_state:202 - [MASSIVE] Begin pre-massive sync hooks for table hive_votes
INFO - hive.db.db_state:202 - [MASSIVE] Begin pre-massive sync hooks for table hive_follows
INFO - hive.db.db_state:202 - [MASSIVE] Begin pre-massive sync hooks for table hive_reblogs
INFO - hive.db.db_state:202 - [MASSIVE] Begin pre-massive sync hooks for table hive_feed_cache
INFO - hive.db.db_state:202 - [MASSIVE] Begin pre-massive sync hooks for table hive_posts_api_helper
INFO - hive.db.db_state:202 - [MASSIVE] Begin pre-massive sync hooks for table hive_communities
INFO - hive.db.db_state:202 - [MASSIVE] Begin pre-massive sync hooks for table hive_subscriptions
INFO - hive.db.db_state:202 - [MASSIVE] Begin pre-massive sync hooks for table hive_notification_cache
INFO - hive.db.db_state:209 - Drop index hivemind_app.hive_follows.hive_follows_created_at_idx
INFO - hive.db.db_state:209 - Drop index hivemind_app.hive_posts.hive_posts_community_id_is_pinned_idx
INFO - hive.db.db_state:209 - Drop index hivemind_app.hive_votes.hive_votes_voter_id_post_id_idx
INFO - hive.db.db_state:209 - Drop index hivemind_app.hive_follows.hive_follows_follower_following_state_idx
INFO - hive.db.db_state:209 - Drop index hivemind_app.hive_follows.hive_follows_following_state_idx
INFO - hive.db.db_state:209 - Drop index hivemind_app.hive_votes.hive_votes_block_num_idx
INFO - hive.db.db_state:209 - Drop index hivemind_app.hive_follows.hive_follows_block_num_idx
INFO - hive.db.db_state:209 - Drop index hivemind_app.hive_posts.hive_posts_block_num_idx
INFO - hive.db.db_state:209 - Drop index hivemind_app.hive_reputation_data.hive_reputation_data_block_num_idx
INFO - hive.db.db_state:209 - Drop index hivemind_app.hive_posts.hive_posts_block_num_created_idx
INFO - hive.db.db_state:209 - Drop index hivemind_app.hive_votes.hive_votes_voter_id_last_update_idx
INFO - hive.db.db_state:209 - Drop index hivemind_app.hive_follows.hive_follows_follower_state_idx
INFO - hive.db.db_state:209 - Drop index hivemind_app.hive_posts.hive_posts_community_id_not_is_pinned_idx
INFO - hive.db.db_state:209 - Drop index hivemind_app.hive_votes.hive_votes_post_id_voter_id_idx
INFO - hive.db.db_state:209 - Drop index hivemind_app.hive_posts.hive_posts_payout_plus_pending_payout_id_idx
INFO - hive.db.db_state:209 - Drop index hivemind_app.hive_reblogs.hive_reblogs_created_at_idx
INFO - hive.db.db_state:209 - Drop index hivemind_app.hive_feed_cache.hive_feed_cache_account_id_created_at_post_id_idx
INFO - hive.db.adapter:90 - Closing database connection: 'hive_follows'
INFO - hive.db.adapter:90 - Closing database connection: 'hive_votes'
INFO - hive.db.db_state:209 - Drop index hivemind_app.hive_posts.hive_posts_cashout_time_id_idx
INFO - hive.db.db_state:232 - [MASSIVE] End pre-massive sync hooks for table hive_votes
INFO - hive.db.db_state:232 - [MASSIVE] End pre-massive sync hooks for table hive_follows
INFO - hive.db.db_state:209 - Drop index hivemind_app.hive_posts.hive_posts_updated_at_idx
INFO - hive.db.db_state:209 - Drop index hivemind_app.hive_communities.hive_communities_block_num_idx
INFO - hive.db.db_state:209 - Drop index hivemind_app.hive_posts_api_helper.hive_posts_api_helper_author_s_permlink_idx
INFO - hive.db.db_state:209 - Drop index hivemind_app.hive_posts.hive_posts_payout_at_idx
INFO - hive.db.adapter:90 - Closing database connection: 'hive_reblogs'
INFO - hive.db.db_state:232 - [MASSIVE] End pre-massive sync hooks for table hive_reblogs
INFO - hive.db.db_state:209 - Drop index hivemind_app.hive_posts.hive_posts_community_id_not_is_paidout_idx
INFO - hive.db.db_state:209 - Drop index hivemind_app.hive_posts.hive_posts_category_id_payout_plus_pending_payout_depth_idx
INFO - hive.db.db_state:209 - Drop index hivemind_app.hive_posts.hive_posts_payout_idx
INFO - hive.db.db_state:209 - Drop index hivemind_app.hive_posts.hive_posts_root_id_id_idx
INFO - hive.db.db_state:209 - Drop index hivemind_app.hive_posts.hive_posts_promoted_id_idx
INFO - hive.db.db_state:209 - Drop index hivemind_app.hive_feed_cache.hive_feed_cache_block_num_idx
INFO - hive.db.adapter:90 - Closing database connection: 'hive_reputation_data'
INFO - hive.db.db_state:209 - Drop index hivemind_app.hive_posts.hive_posts_depth_idx
INFO - hive.db.adapter:90 - Closing database connection: 'hive_posts_api_helper'
INFO - hive.db.adapter:90 - Closing database connection: 'hive_communities'
INFO - hive.db.db_state:209 - Drop index hivemind_app.hive_feed_cache.hive_feed_cache_created_at_idx
INFO - hive.db.db_state:209 - Drop index hivemind_app.hive_posts.hive_posts_parent_id_id_idx
INFO - hive.db.db_state:209 - Drop index hivemind_app.hive_feed_cache.hive_feed_cache_post_id_idx
INFO - hive.db.db_state:232 - [MASSIVE] End pre-massive sync hooks for table hive_posts_api_helper
INFO - hive.db.db_state:232 - [MASSIVE] End pre-massive sync hooks for table hive_communities
INFO - hive.db.db_state:232 - [MASSIVE] End pre-massive sync hooks for table hive_reputation_data
INFO - hive.db.db_state:209 - Drop index hivemind_app.hive_posts.hive_posts_sc_trend_id_idx
INFO - hive.db.adapter:90 - Closing database connection: 'hive_feed_cache'
INFO - hive.db.db_state:209 - Drop index hivemind_app.hive_notification_cache.hive_notification_cache_dst_score_idx
INFO - hive.db.db_state:232 - [MASSIVE] End pre-massive sync hooks for table hive_feed_cache
INFO - hive.db.db_state:209 - Drop index hivemind_app.hive_posts.hive_posts_tags_ids_idx
INFO - hive.db.db_state:209 - Drop index hivemind_app.hive_posts.hive_posts_sc_hot_id_idx
INFO - hive.db.db_state:209 - Drop index hivemind_app.hive_posts.hive_posts_author_id_created_at_id_idx
INFO - hive.db.db_state:209 - Drop index hivemind_app.hive_posts.hive_posts_community_id_id_idx
INFO - hive.db.db_state:209 - Drop index hivemind_app.hive_subscriptions.hive_subscriptions_community_idx
INFO - hive.db.db_state:209 - Drop index hivemind_app.hive_posts.hive_posts_author_id_id_idx
INFO - hive.db.adapter:90 - Closing database connection: 'hive_posts'
INFO - hive.db.db_state:232 - [MASSIVE] End pre-massive sync hooks for table hive_posts
INFO - hive.db.db_state:209 - Drop index hivemind_app.hive_subscriptions.hive_subscriptions_block_num_idx
INFO - hive.db.adapter:90 - Closing database connection: 'hive_subscriptions'
INFO - hive.db.db_state:232 - [MASSIVE] End pre-massive sync hooks for table hive_subscriptions
INFO - hive.db.db_state:209 - Drop index hivemind_app.hive_notification_cache.hive_notification_cache_block_num_idx
INFO - hive.db.adapter:90 - Closing database connection: 'hive_notification_cache'
INFO - hive.db.db_state:232 - [MASSIVE] End pre-massive sync hooks for table hive_notification_cache
INFO - hive.db.db_state:417 - [MASSIVE] 10 threads finished creating indexes. Real elapsed time: 0.469
INFO - hive.db.db_state:254 - === DROPPING INDEXES ===
INFO - hive.utils.stats:162 - ####################
INFO - hive.utils.stats:300 - Total DROPPING indexes time
INFO - hive.utils.stats:156 - `hive_notification_cache`: Processed final operations in 0.4109 seconds
INFO - hive.utils.stats:156 - `hive_posts`: Processed final operations in 0.4028 seconds
INFO - hive.utils.stats:156 - `hive_subscriptions`: Processed final operations in 0.3963 seconds
INFO - hive.utils.stats:156 - `hive_reputation_data`: Processed final operations in 0.3575 seconds
INFO - hive.utils.stats:156 - `hive_feed_cache`: Processed final operations in 0.3446 seconds
INFO - hive.utils.stats:156 - `hive_posts_api_helper`: Processed final operations in 0.3271 seconds
INFO - hive.utils.stats:156 - `hive_communities`: Processed final operations in 0.3232 seconds
INFO - hive.utils.stats:156 - `hive_reblogs`: Processed final operations in 0.2838 seconds
INFO - hive.utils.stats:156 - `hive_votes`: Processed final operations in 0.2748 seconds
INFO - hive.utils.stats:156 - `hive_follows`: Processed final operations in 0.2723 seconds
INFO - hive.utils.stats:302 - Current final processing time: 3.3932s.
INFO - hive.db.db_state:256 - Elapsed time: 0.5356s. Calculated elapsed time: 3.3932s. Difference: -2.8577s
INFO - hive.db.db_state:260 - === DROPPING INDEXES ===
INFO - hive.db.db_state:278 - Dropping FKs
INFO - hive.db.db_state:284 - [MASSIVE] Finish pre-massive sync hooks
INFO - hive.indexer.hive_db.haf_functions:25 - Trying to detach app context...
INFO - hive.indexer.hive_db.haf_functions:27 - App context detaching done.
INFO - hive.indexer.sync:137 - [MASSIVE] Attempting to process block range: <28001:7406908>
INFO - hive.indexer.blocks:142 - End-of-sync LIB is set to 7406908, last block that guarantees cashout at end of sync is 0
INFO - hive.indexer.hive_db.massive_blocks_data_provider:266 - blocks_data_queue.qsize: 0
INFO - hive.indexer.hive_db.massive_blocks_data_provider:267 - operations_queue.qsize: 0
INFO - hive.indexer.hive_db.massive_blocks_data_provider:268 - blocks_queue.qsize: 0
INFO - hive.indexer.hive_db.massive_blocks_data_provider:274 - Awaiting any blocks to process... 0

changed the description

1. poor sync performance is caused by filling up HAF database and thus having dropped indexes (to have better write performance). Because of that, Hivemind (which uses HAF data in really complex way, by filtering operations basing on block range and operation type conditions) is unable to get them in acceptable performance level. To have it working well, just complete first setup of HAF instance, including building its indexes, and next start Hivemind sync process.
2. Because in dockerized setup run_instance.sh <docker_img> replaces hive shell wrapper, to start sync or server process, you should pass sync or server phrase after run_instance. Consider example from README.md:

$ ../hivemind/scripts/run_instance.sh registry.gitlab.syncad.com/hive/hivemind/instance:local sync --database-url="postgresql://haf_app_admin@172.17.0.2:5432/haf_block_log"

@bwrona Thanks, that worked.

1. When running the Hivemind sync after filling up the HAF db was done, the sync was super fast.
2. OK, I didn't realize that wrapper command could be changed to use the server phrase. It worked. Maybe something to clarify in the docs (I would submit a PR but not too confident yet in the correctness of my understanding).

closed

Yes it has a historical reasons that hivemind is splitted into 2 processes:

• one serving API calls (server)
• second sync ing the database (shared with server process) Originally, they are started by specifying hive sync or hive server commands (when operating directly in python env). But when we're using dockerized setup, hive command is somehow replaced by run_instance.sh call which next takes other params.

Feel free to prepare MR updating docs.

@bwrona Gitlab says I can't create a fork. And I don't have access to directly make an MR to the repo either.