hmind server startup needs to wait for blockprocessing to start
It looks like maybe the blockprocessing docker recreates the hivemind schema, leading to a potential race that can cause hivemind server to die during its startup. Here is log from hivemind blockprocessing:
2024-01-17T04:09:02.962377641Z [Entrypoint] 2024-01-17 04:09:02,962+00:00 INFO [global] (main) Hivemind arguments: --database-url=postgresql://hivemind@haf/ haf_block_log sync
2024-01-17T04:09:02.963084663Z [Entrypoint] 2024-01-17 04:09:02,962+00:00 INFO [global] (main) Hivemind arguments: sync
2024-01-17T04:09:02.963562165Z [Entrypoint] 2024-01-17 04:09:02,963+00:00 INFO [global] (main) Using PostgreSQL instance: postgresql://hivemind@haf/ haf_block_log
2024-01-17T04:09:02.964499218Z [Entrypoint] 2024-01-17 04:09:02,964+00:00 INFO [run_hive] (main) Starting Hivemind...
2024-01-17T04:09:05.443593327Z INFO - hive.conf:212 - The database instance is created...
2024-01-17T04:09:05.444564331Z INFO - hive.db.adapter:48 - A database offers maximum connections: 100. Required 15 connections.
2024-01-17T04:09:05.444574411Z INFO - hive.db.adapter:90 - Closing database connection: 'root'
2024-01-17T04:09:05.444836781Z INFO - hive.db.adapter:108 - Disposing SQL engine
2024-01-17T04:09:05.444924532Z INFO - hive.conf:261 - The database is disconnected...
2024-01-17T04:09:05.449229476Z INFO - hive.conf:212 - The database instance is created...
2024-01-17T04:09:05.920817366Z INFO - hive.indexer.sync:52 - Entering HAF mode synchronization
2024-01-17T04:09:05.920855406Z INFO - hive.db.db_state:44 - [MASSIVE] Welcome to hive!
2024-01-17T04:09:05.922196891Z INFO - hive.db.db_state:48 - [MASSIVE] Create db schema...
And here is fail case for hivemind server:
2024-01-17T04:09:05.927718829Z
WARNING - hive.server.db:111 - [SQL-ERR] InvalidSchemaName in query SELECT num FROM hivemind_app.get_head_state(); ({})
ERROR - asyncio:1707 - unhandled exception during asyncio.run() shutdown
task: <Task finished name='Task-1' coro=<_run_app() done, defined at /home/hivemind/.hivemind-venv/lib/python3.8
/site-packages/aiohttp/web.py:287> exception=InvalidSchemaName('schema "hivemind_app" does not exist\nLINE 1: SELECT num FROM hivemind_app.get_head_state();\n ^\n')>
Traceback (most recent call last):
File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/aiohttp/web.py", line 514, in run_app
loop.run_until_complete(main_task)
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/aiohttp/web.py", line 321, in _run_app
await runner.setup()
File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/aiohttp/web_runner.py", line 279, in setup
self._server = await self._make_server()
File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/aiohttp/web_runner.py", line 375, in _make_server
await self._app.startup()
File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/aiohttp/web_app.py", line 417, in startup
await self.on_startup.send(self)
File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/aiosignal/__init__.py", line 36, in send
await receiver(*args, **kwargs) # type: ignore
File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/hive/server/serve.py", line 251, in show_info
last = await app['db'].query_one(f"SELECT num FROM {SCHEMA_NAME}.get_head_state();")
File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/hive/server/db.py", line 21, in _wrapper
result = await function(*args, **kwargs)
File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/hive/server/db.py", line 96, in query_one
cur = await self._query(conn, sql, **kwargs)
File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/hive/server/db.py", line 112, in _query
Edited by Dan Notestein