Sync crashes after finishing massive sync. Some connections not closed.

I've been having this issue of my hivemind sync seemingly disappearing after a while for a while now. Having learnt that it auto removed on close it made it difficult to figure out why

I've finaly had it crash/close while watching the logs and now know why it's not working.

Not sure what to do about it though, I have rarely seen my node enter single block mode either because it's to slow to keep up (usualy due to reputation calcualtions taking too long) or it just closes likely with the below error.

Update: Using HAF 1.27.4 and HAF Hivemind 1.27.4 I've attached my config.ini as requested by blocktrades (sensitive data omitted ofcourse).

Update 2: VM:

running 8 vcores and 128GB ram
Storage type, Software Raid0 (host) Enterprise SSDs x2, 4TB each.
These drives only serve the HIVE VM.
Haf, Hafah and Haf Hivemind all run on the same VM. All are run as docker containers.

Host machine:

Proxmox base OS
CPU: Xeon E5-2690 v2 x2 (2 Sockets)
Ram: 256GB ECC DDR3
MOBO: ASUS Z9PE-D8 WS

Other VMs running:

small reverse proxy feeding data to hive and various other endpoints
A family media server
A game server for satisfactory
and a small VM for a friend.

All these a rather light weight and barely dent the server resources. None of these VMs are on the same drives as HIVE.

Update 3:

Some addition and perhaps relevant info. On my server in it's current state of not going to single block mode, massive sync is usually processing around 140 blocks with each 'pass' so to say. Each batch of blocks it works on takes long enough that there are 140 odd blocks to process again. and this loops for a number of times until it inevitably throws the error below.

It can run this way anywhere from 15mins to several hours. Recently it's mostly been lasting 30mins to and hour before dying. (I've been sat with a console open watching the logs go by as I do other stuff.


INFO - hive.db.db_state:520 - [MASSIVE] After massive sync actions done in 4456.2228s
INFO - hive.db.db_state:76 - [MASSIVE] Massive sync complete!
INFO - hive.indexer.hive_db.haf_functions:37 - Trying to attach app context with block number: 77797737
INFO - hive.indexer.hive_db.haf_functions:39 - App context attaching done.
INFO - hive.db.adapter:90 - Closing database connection: 'PostDataCache'
INFO - hive.db.adapter:90 - Closing database connection: 'Reputations'
INFO - hive.db.adapter:90 - Closing database connection: 'Votes'
INFO - hive.db.adapter:90 - Closing database connection: 'Follow'
INFO - hive.db.adapter:90 - Closing database connection: 'Posts'
INFO - hive.db.adapter:90 - Closing database connection: 'Reblog'
INFO - hive.db.adapter:90 - Closing database connection: 'Notify'
INFO - hive.db.adapter:90 - Closing database connection: 'Accounts'
INFO - hive.db.adapter:90 - Closing database connection: 'PayoutStats'
INFO - hive.db.adapter:90 - Closing database connection: 'Mentions'
INFO - hive.db.adapter:90 - Closing database connection: 'MassiveBlocksProvider_OperationsData'
INFO - hive.db.adapter:90 - Closing database connection: 'MassiveBlocksProvider_BlocksData'
INFO - hive.indexer.sync:71 - Exiting HAF mode synchronization
WARNING - hive.server.common.payout_stats:16 - Rebuilding payout_stats_view in separate transaction
INFO - hive.indexer.sync:76 - LAST IMPORTED BLOCK IS: 77797737
INFO - hive.indexer.sync:77 - LAST COMPLETED BLOCK IS: 77797737
INFO - hive.indexer.hive_db.haf_functions:34 - Context already attached - attaching skipped.
INFO - hive.db.adapter:90 - Closing database connection: 'PostDataCache'
INFO - hive.db.adapter:90 - Closing database connection: 'Reputations'
INFO - hive.db.adapter:90 - Closing database connection: 'Votes'
INFO - hive.db.adapter:90 - Closing database connection: 'Follow'
INFO - hive.db.adapter:90 - Closing database connection: 'Posts'
INFO - hive.db.adapter:90 - Closing database connection: 'Reblog'
INFO - hive.db.adapter:90 - Closing database connection: 'Notify'
INFO - hive.db.adapter:90 - Closing database connection: 'Accounts'
INFO - hive.db.adapter:90 - Closing database connection: 'PayoutStats'
INFO - hive.db.adapter:90 - Closing database connection: 'Mentions'
INFO - hive.db.adapter:90 - Closing database connection: 'root'
INFO - hive.db.adapter:103 - Disposing SQL engine
INFO - hive.conf:254 - The database is disconnected...
Traceback (most recent call last):
  File "/home/hivemind/.hivemind-venv/bin/hive", line 8, in 
    sys.exit(run())
  File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/hive/cli.py", line 73, in run
    launch_mode(mode, conf)
  File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/hive/cli.py", line 87, in launch_mode
    sync.run()
  File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/hive/indexer/sync.py", line 151, in run
    self._assert_connections_closed(active_connections_before, active_connections_after_massive)
  File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/hive/indexer/sync.py", line 366, in _assert_connections_closed
    assert set(connections_before) == set(connections_after), assert_message
AssertionError: Some db connections used in MASSIVE sync were not closed!
before: [('hivemind_root',), ('hivemind_root',)]
after: [('hivemind_root',), ('hivemind_root',), ('hivemind_PayoutStats',)]
Exiting docker entrypoint...

config.ini

Haf Log:haflog.zip The hivemind sync failed sometime between 2AM 1/09 and 2:45AM

Edited Sep 01, 2023 by Ben Swinburn