GitLab

On a few occasions hivemind froze after hitting a fork too deep error. When frozen, hivemind doen't process any requests.

hive-hivemind | Traceback (most recent call last):
hive-hivemind |   File "/usr/local/bin/hive", line 8, in <module>
hive-hivemind |     sys.exit(run())
hive-hivemind |   File "/usr/local/lib/python3.6/dist-packages/hive/cli.py", line 23, in run
hive-hivemind |     launch_mode(mode, conf)
hive-hivemind |   File "/usr/local/lib/python3.6/dist-packages/hive/cli.py", line 34, in launch_mode
hive-hivemind |     Sync(conf=conf).run()
hive-hivemind |   File "/usr/local/lib/python3.6/dist-packages/hive/indexer/sync.py", line 64, in run
hive-hivemind |     Blocks.verify_head(self._steem)
hive-hivemind |   File "/usr/local/lib/python3.6/dist-packages/hive/indexer/blocks.py", line 129, in verify_head
hive-hivemind |     assert hive_head - cursor < 25, "fork too deep"
hive-hivemind | AssertionError: fork too deep

A workaround is to log into the postgres database and delete ~10000 lines before the last processed block, for example:

ALTER TABLE hive_blocks DISABLE TRIGGER ALL;
DELETE FROM hive_blocks WHERE num > 42200000;
ALTER TABLE hive_blocks ENABLE TRIGGER ALL;

Then restart hivemind to let it synch.

The DISABLE/ENABLE TRIGGER is needed otherwise deleting multiple lines is super slow! (https://dba.stackexchange.com/a/37095/180936)

Suggested solution: Implement this temporary workaround in the code. Or find another way to handle that error.