Fresh synch frozen, exception encountered block 45152664
This is the n-th time I've tried to get hivemind working in a docker environment, everytime there's something wrong with it. This time, it's stuck, again.
Using deployed_20200928_pure
, started a fresh new synch from scratch, entered the docker container and ran the db_update.sh script. Synching went well, then days later it froze at head block 45152000.
{
"state": {
"db_head_block": 45152000,
"db_head_time": "2020-07-14 17:07:03",
"db_head_age": 8981670
},
"result": "head block age (8981670) > max (3600); head block num: 45152000",
"status": "WARN",
"sync_service": false,
"source_commit": "296a91050dff8790dc52019f9ea203a316cb7951",
"schema_hash": "adb5cd9b",
"docker_tag": "deployed_20200928_pure",
"timestamp": "2020-10-26T16:01:33.392637"
}
Restarted hivemind container and got this log, exception encountered at block 45152664 and froze:
hivemind | postgres:5432 - accepting connections
hivemind | hivemind: starting sync
hivemind | hivemind: starting server
hivemind | INFO:root:Used command line args: server
hivemind | INFO:root:Used command line args: sync
hivemind | INFO:hive.indexer.sync:Using hived url: `{"default":"http://192.168.2.200:8090"}'
hivemind | INFO:hive.steem.client:Endpoint default will be routed to node http://192.168.2.200:8090
hivemind | INFO:hive.steem.http_client:using node: http://192.168.2.200:8090
hivemind | INFO:hive.indexer.sync:hivemind_version : 0.0.1
hivemind | INFO:hive.indexer.sync:hivemind_git_rev : 296a910
hivemind | INFO:hive.indexer.sync:database_schema_version : 18
hivemind | INFO:hive.db.db_state:[INIT] Welcome to hive!
hivemind | WARNING:aiocache.serializers.serializers:msgpack not installed, MsgPackSerializer unavailable
hivemind | INFO:aiocache:aioredis not installed, RedisCache unavailable
hivemind | INFO:aiocache:aiomcache not installed, Memcached unavailable
hivemind | INFO:hive.db.db_state:[HIVE] db version: 18
hivemind | INFO:hive.db.db_state:[INIT] Continue with initial sync...
hivemind | INFO:hive.server.serve:hivemind_version : 0.0.1
hivemind | INFO:hive.server.serve:hivemind_git_rev : 296a910
hivemind | INFO:hive.server.serve:database_schema_version : 18
hivemind | INFO:hive.server.serve:database_head_block : 45152000
hivemind | INFO:hive.indexer.sync:database_head_block : 45152000
hivemind | INFO:hive.indexer.sync:target_head_block : 48137869
hivemind | INFO:hive.db.db_state:[INIT] Begin pre-initial sync hooks
hivemind | INFO:hive.db.db_state:Drop index hive_reputation_data.hive_reputation_data_block_num_idx
hivemind | INFO:hive.db.db_state:Drop index hive_posts.hive_posts_payout_idx
hivemind | INFO:hive.db.db_state:Drop index hive_posts.hive_posts_payout_plus_pending_payout_id_is_paidout_idx
hivemind | INFO:hive.db.db_state:Drop index hive_posts.hive_posts_parent_id_idx
hivemind | INFO:hive.db.db_state:Drop index hive_posts.hive_posts_promoted_idx
hivemind | INFO:hive.db.db_state:Drop index hive_posts.hive_posts_block_num_idx
hivemind | INFO:hive.db.db_state:Drop index hive_posts.hive_posts_community_id_idx
hivemind | INFO:hive.db.db_state:Drop index hive_posts.hive_posts_sc_trend_id_is_paidout_idx
hivemind | INFO:hive.db.db_state:Drop index hive_posts.hive_posts_cashout_time_id_idx
hivemind | INFO:hive.db.db_state:Drop index hive_posts.hive_posts_category_id_idx
hivemind | INFO:hive.db.db_state:Drop index hive_posts.hive_posts_sc_hot_id_is_paidout_idx
hivemind | INFO:hive.db.db_state:Drop index hive_posts.hive_posts_depth_idx
hivemind | INFO:hive.db.db_state:Drop index hive_posts.hive_posts_updated_at_idx
hivemind | INFO:hive.db.db_state:Drop index hive_posts.hive_posts_payout_at_idx
hivemind | INFO:hive.db.db_state:Drop index hive_posts.hive_posts_created_at_idx
hivemind | INFO:hive.db.db_state:Drop index hive_posts.hive_posts_root_id_id_idx
hivemind | INFO:hive.db.db_state:Drop index hive_votes.hive_votes_post_id_voter_id_idx
hivemind | INFO:hive.db.db_state:Drop index hive_votes.hive_votes_voter_id_post_id_idx
hivemind | INFO:hive.db.db_state:Drop index hive_votes.hive_votes_block_num_idx
hivemind | INFO:hive.db.db_state:Drop index hive_post_tags.hive_post_tags_tag_id_idx
hivemind | INFO:hive.db.db_state:Drop index hive_follows.hive_follows_ix5a
hivemind | INFO:hive.db.db_state:Drop index hive_follows.hive_follows_created_at_idx
hivemind | INFO:hive.db.db_state:Drop index hive_follows.hive_follows_ix5b
hivemind | INFO:hive.db.db_state:Drop index hive_reblogs.hive_reblogs_created_at_idx
hivemind | INFO:hive.db.db_state:Drop index hive_communities.hive_communities_block_num_idx
hivemind | INFO:hive.db.db_state:Drop index hive_subscriptions.hive_subscriptions_community_idx
hivemind | INFO:hive.db.db_state:Drop index hive_subscriptions.hive_subscriptions_block_num_idx
hivemind | INFO:hive.db.db_state:Dropping FKs
hivemind | INFO:hive.db.db_state:[INIT] Finish pre-initial sync hooks
hivemind | INFO:hive.indexer.sync:[INIT] *** Initial fast sync ***
hivemind | INFO:hive.indexer.sync:[SYNC] start block 45152001, +2985869 to sync
hivemind | INFO:hive.indexer.sync:[SYNC] start vops 45152001, +2985869 to sync
hivemind | INFO:hive.indexer.sync:Awaiting any block to process...
hivemind | INFO:hive.indexer.sync:Awaiting any vops to process...
hivemind | WARNING:hive.steem.http_client:get_block[50] failed in 0.0s. try 1. {'secs': 0.001, 'try': 1} - ProtocolError('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))
hivemind | WARNING:hive.steem.http_client:get_block[50] failed in 0.0s. try 1. {'secs': 0.001, 'try': 1} - ProtocolError('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))
hivemind | INFO:hive.indexer.community:validating @mattsanthonyit op ['subscribe', {'community': 'hive-114929'}]
hivemind | WARNING:hive.indexer.custom_op:notify op fail: in ['setLastRead', {'date': '2020-07-14T17:11:00'}]
hivemind | WARNING:hive.indexer.custom_op:unexpected auths: {'required_auths': [], 'required_posting_auths': ['certain', 'threespeak'], 'id': '3speak-publish', 'json': '{"author":"certain","permlink":"xmgxvvej","category":"general","language":"en","duration":299.096235,"title":"In the end, I was successful in defeating Corona. Thanks God my corona test report came negative.."}', 'block_num': 45152063}
hivemind | WARNING:hive.indexer.custom_op:unexpected auths: {'required_auths': [], 'required_posting_auths': ['schneidy', 'threespeak'], 'id': '3speak-publish', 'json': '{"author":"schneidy","permlink":"rjnxaaqm","category":"general","language":"en","duration":44.033333,"title":"5th day of 25 push ups challenge (Watch until the end for doggie kisses)"}', 'block_num': 45152065}
hivemind | INFO:hive.indexer.community:validating @hive-108278 op ['setUserTitle', {'community': 'hive-108278', 'account': 'globetrottergcc', 'title': "Official star's community member"}]
hivemind | INFO:hive.indexer.community:validating @rieradg op ['subscribe', {'community': 'hive-174301'}]
hivemind | INFO:hive.indexer.community:validating @josemoises op ['subscribe', {'community': 'hive-148441'}]
hivemind | INFO:hive.indexer.community:validating @josemoises op ['subscribe', {'community': 'hive-174578'}]
hivemind | INFO:hive.indexer.community:validating @josemoises op ['subscribe', {'community': 'hive-156509'}]
hivemind | INFO:hive.indexer.community:validating @josemoises op ['subscribe', {'community': 'hive-196037'}]
hivemind | INFO:hive.indexer.community:validating @josemoises op ['subscribe', {'community': 'hive-174301'}]
hivemind | INFO:hive.indexer.community:validating @hive-108278 op ['setRole', {'community': 'hive-108278', 'account': 'globetrottergcc', 'role': 'member'}]
hivemind | INFO:hive.indexer.community:validating @josemoises op ['subscribe', {'community': 'hive-133872'}]
hivemind | INFO:hive.indexer.community:validating @josemoises op ['subscribe', {'community': 'hive-158694'}]
hivemind | INFO:hive.indexer.community:validating @mattsanthonyit op ['subscribe', {'community': 'hive-152889'}]
hivemind | INFO:hive.indexer.community:validating @josemoises op ['subscribe', {'community': 'hive-192870'}]
hivemind | INFO:hive.indexer.community:validating @josemoises op ['subscribe', {'community': 'hive-193816'}]
hivemind | INFO:hive.indexer.community:validating @josemoises op ['subscribe', {'community': 'hive-102880'}]
hivemind | INFO:hive.indexer.community:validating @josemoises op ['subscribe', {'community': 'hive-186377'}]
hivemind | INFO:hive.indexer.community:validating @josemoises op ['subscribe', {'community': 'hive-105786'}]
hivemind | INFO:hive.indexer.community:validating @mattsanthonyit op ['subscribe', {'community': 'hive-122136'}]
hivemind | INFO:hive.indexer.community:validating @josemoises op ['subscribe', {'community': 'hive-182074'}]
hivemind | INFO:hive.indexer.community:validating @josemoises op ['subscribe', {'community': 'hive-110490'}]
hivemind | INFO:hive.indexer.community:validating @josemoises op ['subscribe', {'community': 'hive-189678'}]
hivemind | WARNING:hive.indexer.custom_op:notify op fail: in ['setLastRead', {'date': '2020-07-14T17:47:09'}]
hivemind | WARNING:hive.indexer.custom_op:notify op fail: in ['setLastRead', {'date': '2020-07-14T17:22:40'}]
hivemind | INFO:hive.indexer.community:validating @hive-102502 op ['updateProps', {'community': 'hive-102502', 'props': {'title': 'Pixelart', 'about': 'A creative community for original pixelart creations.', 'lang': 'en', 'description': "Only post your original pixelart.\nNon traditional forms of pixelart are welcome- for example- perler beads, but only original creations, you can't use other artists' art for reference. \nVoxel art and styles are allowed, I hope there is will be another community for it eventually. \nYou can post about pixelart programs and tutorials on how to use them.\n\nIf you need a hive account use https://hiveonboard.com/", 'flag_text': "No plagarism\nOnly pixelart posts\nOriginal work only\nAvoid copies/studies of other art work (Studies of old classic art is allowed- for example: Mona Lisa)\nNon-traditional pixelart is welcome \nCross-Posts are welcome\nReposts are welcome\nNo filters- (like auto pixelate programs)\nDon't be rude and use constructive criticism \nPopular characters are allowed only if it's your original take on the character (for example Pokemons- don't redraw already done art, only your original interpretation)", 'is_nsfw': False}}]
hivemind | WARNING:hive.indexer.custom_op:notify op fail: in ['setLastRead', {'date': '2020-07-14T17:24:58'}]
hivemind | WARNING:hive.indexer.custom_op:notify op fail: in ['setLastRead', {'date': '2020-07-14T17:29:51'}]
hivemind | WARNING:hive.indexer.custom_op:unexpected auths: {'required_auths': [], 'required_posting_auths': ['lisamgentile1961', 'threespeak'], 'id': '3speak-publish', 'json': '{"author":"lisamgentile1961","permlink":"vmymjatn","category":"general","language":"en","duration":315.147938,"title":"Don\'t Just Send Me A Message To Pitch Me! || I Am Alive On Hive Challenge Day 97 #IAmAliveChallenge - #HiveIsAlive"}', 'block_num': 45152462}
hivemind | INFO:hive.indexer.community:validating @violetmed op ['subscribe', {'community': 'hive-161155'}]
hivemind | WARNING:hive.indexer.custom_op:notify op fail: in ['setLastRead', {'date': '2020-07-14T17:34:25'}]
hivemind | INFO:hive.indexer.community:validating @rorrochileno op ['subscribe', {'community': 'hive-196037'}]
hivemind | WARNING:hive.indexer.custom_op:unexpected auths: {'required_auths': [], 'required_posting_auths': ['badpupper', 'threespeak'], 'id': '3speak-publish', 'json': '{"author":"badpupper","permlink":"kwxqzwuy","category":"general","language":"en","duration":662.717732,"title":"The Gnoblin King and his Scary Men [13]"}', 'block_num': 45152561}
hivemind | WARNING:hive.indexer.custom_op:notify op fail: in ['setLastRead', {'date': '2020-07-14T17:35:40'}]
hivemind | WARNING:hive.indexer.custom_op:notify op fail: in ['setLastRead', {'date': '2020-07-14T17:39:58'}]
hivemind | WARNING:hive.db.adapter:[SQL-ERR] ProgrammingError in query select id from hive_accounts where name in :names ({'names': ()})
hivemind | ERROR:hive.indexer.blocks:exception encountered block 45152664
hivemind | ERROR:hive.indexer.sync:Exception caught during processing blocks...
hivemind | Traceback (most recent call last):
hivemind | File "/root/.local/lib/python3.6/site-packages/SQLAlchemy-1.3.20-py3.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1277, in _execute_context
hivemind | cursor, statement, parameters, context
hivemind | File "/root/.local/lib/python3.6/site-packages/SQLAlchemy-1.3.20-py3.6-linux-x86_64.egg/sqlalchemy/engine/default.py", line 593, in do_execute
hivemind | cursor.execute(statement, parameters)
hivemind | psycopg2.errors.SyntaxError: syntax error at or near ")"
hivemind | LINE 1: select id from hive_accounts where name in ()
hivemind | ^
hivemind |
hivemind |
hivemind | The above exception was the direct cause of the following exception:
hivemind |
hivemind | Traceback (most recent call last):
hivemind | File "/root/.local/lib/python3.6/site-packages/hivemind-0.0.1+296a910-py3.6.egg/hive/indexer/sync.py", line 137, in _block_consumer
hivemind | Blocks.process_multi(blocks, preparedVops, is_initial_sync)
hivemind | File "/root/.local/lib/python3.6/site-packages/hivemind-0.0.1+296a910-py3.6.egg/hive/indexer/blocks.py", line 110, in process_multi
hivemind | raise e
hivemind | File "/root/.local/lib/python3.6/site-packages/hivemind-0.0.1+296a910-py3.6.egg/hive/indexer/blocks.py", line 107, in process_multi
hivemind | last_num = cls._process(block, vops)
hivemind | File "/root/.local/lib/python3.6/site-packages/hivemind-0.0.1+296a910-py3.6.egg/hive/indexer/blocks.py", line 287, in _process
hivemind | CustomOp.process_ops(json_ops, num, cls._head_block_date)
hivemind | File "/root/.local/lib/python3.6/site-packages/hivemind-0.0.1+296a910-py3.6.egg/hive/indexer/custom_op.py", line 54, in process_ops
hivemind | cls._process_legacy(account, op_json, block_date, block_num)
hivemind | File "/root/.local/lib/python3.6/site-packages/hivemind-0.0.1+296a910-py3.6.egg/hive/indexer/custom_op.py", line 106, in _process_legacy
hivemind | Follow.follow_op(account, op_json, block_date, block_num)
hivemind | File "/root/.local/lib/python3.6/site-packages/hivemind-0.0.1+296a910-py3.6.egg/hive/indexer/follow.py", line 93, in follow_op
hivemind | op = cls._validated_op(account, op_json, date)
hivemind | File "/root/.local/lib/python3.6/site-packages/hivemind-0.0.1+296a910-py3.6.egg/hive/indexer/follow.py", line 167, in _validated_op
hivemind | flg=Accounts.get_id(op['following']) if not isinstance(op['following'], list) else cls._get_ids_for_accounts(op['following']),
hivemind | File "/root/.local/lib/python3.6/site-packages/hivemind-0.0.1+296a910-py3.6.egg/hive/indexer/follow.py", line 393, in _get_ids_for_accounts
hivemind | sql_result = DB.query_all(sql, names=tuple(accounts))
hivemind | File "/root/.local/lib/python3.6/site-packages/hivemind-0.0.1+296a910-py3.6.egg/hive/db/adapter.py", line 96, in query_all
hivemind | res = self._query(sql, **kwargs)
hivemind | File "/root/.local/lib/python3.6/site-packages/hivemind-0.0.1+296a910-py3.6.egg/hive/db/adapter.py", line 200, in _query
hivemind | raise e
hivemind | File "/root/.local/lib/python3.6/site-packages/hivemind-0.0.1+296a910-py3.6.egg/hive/db/adapter.py", line 192, in _query
hivemind | result = self._exec(query, **kwargs)
hivemind | File "/root/.local/lib/python3.6/site-packages/SQLAlchemy-1.3.20-py3.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1011, in execute
hivemind | return meth(self, multiparams, params)
hivemind | File "/root/.local/lib/python3.6/site-packages/SQLAlchemy-1.3.20-py3.6-linux-x86_64.egg/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
hivemind | return connection._execute_clauseelement(self, multiparams, params)
hivemind | File "/root/.local/lib/python3.6/site-packages/SQLAlchemy-1.3.20-py3.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1130, in _execute_clauseelement
hivemind | distilled_params,
hivemind | File "/root/.local/lib/python3.6/site-packages/SQLAlchemy-1.3.20-py3.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1317, in _execute_context
hivemind | e, statement, parameters, cursor, context
hivemind | File "/root/.local/lib/python3.6/site-packages/SQLAlchemy-1.3.20-py3.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1511, in _handle_dbapi_exception
hivemind | sqlalchemy_exception, with_traceback=exc_info[2], from_=e
hivemind | File "/root/.local/lib/python3.6/site-packages/SQLAlchemy-1.3.20-py3.6-linux-x86_64.egg/sqlalchemy/util/compat.py", line 182, in raise_
hivemind | raise exception
hivemind | File "/root/.local/lib/python3.6/site-packages/SQLAlchemy-1.3.20-py3.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1277, in _execute_context
hivemind | cursor, statement, parameters, context
hivemind | File "/root/.local/lib/python3.6/site-packages/SQLAlchemy-1.3.20-py3.6-linux-x86_64.egg/sqlalchemy/engine/default.py", line 593, in do_execute
hivemind | cursor.execute(statement, parameters)
hivemind | sqlalchemy.exc.ProgrammingError: (psycopg2.errors.SyntaxError) syntax error at or near ")"
hivemind | LINE 1: select id from hive_accounts where name in ()
hivemind | ^
hivemind |
hivemind | [SQL: select id from hive_accounts where name in %(names)s]
hivemind | [parameters: {'names': ()}]
hivemind | (Background on this error at: http://sqlalche.me/e/13/f405)
hivemind | INFO:hive.indexer.sync:=== TOTAL STATS ===
hivemind | INFO:hive.utils.stats:####################
hivemind | INFO:hive.utils.stats:Total waiting times
hivemind | INFO:hive.utils.stats:Total waiting time: 0.0000s.
hivemind | INFO:hive.utils.stats:####################
hivemind | INFO:hive.utils.stats:Total flush times
hivemind | INFO:hive.utils.stats:Total flushing time: 0.0000s.
hivemind | INFO:hive.utils.stats:####################
hivemind | INFO:hive.utils.stats:All operations present in the processed blocks
hivemind | INFO:hive.utils.stats:Total time for processing operations time: 0.0000s.
hivemind | INFO:hive.indexer.sync:Elapsed time: 12.8815s. Calculated elapsed time: 0.0000s. Difference: 12.8815s
Trying to check curl http://localhost:8000/health
becomes unresponsive. Even going into the container and running hive status
is unresponsive too.