Hivemind crashed on startup on "clean" system
Reported by mahdiyari:
What I did:
clone haf_api_node
create zfs datasets
edit .env to --replay-blockchain
copied over block_log and artifacts to blockchain folder
docker compose up -d
haf is replaying just fine
docker-ps-a.txt - Text
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
840e41b152f3 registry.hive.blog/hivemind/instance:v1.27.5rc7 "/home/hivemind/dock…" 13 minutes ago Created haf-world-hivemind-server-1
64153c123f64 registry.gitlab.syncad.com/hive/haf_api_node/postgrest:latest "/bin/postgrest" 13 minutes ago Up 13 minutes (healthy) haf-world-hafah-postgrest-1
82cf6cdeee0e registry.hive.blog/jussi:latest "python -m jussi.ser…" 13 minutes ago Exited (0) 11 minutes ago haf-world-jussi-1
8d37f7ae33b2 registry.hive.blog/hivemind/instance:v1.27.5rc7 "/home/hivemind/dock…" 13 minutes ago Exited (1) 11 minutes ago haf-world-hivemind-block-processing-1
734728a5e946 ankane/pghero:v3.3.3 "/bin/sh -c 'puma -C…" 13 minutes ago Up 13 minutes 8080/tcp haf-world-pghero-1
19124fe9acfd registry.hive.blog/hafah/setup:v1.27.5rc7 "/hafah/scripts/setu…" 13 minutes ago Exited (0) 11 minutes ago haf-world-hafah-install-1
da6309a1e8db varnish:7.3.0-alpine "/usr/local/bin/dock…" 13 minutes ago Up 13 minutes (healthy) 80/tcp, 8443/tcp haf-world-varnish-1
26d0e42e8ff9 redis:7.2-alpine "docker-entrypoint.s…" 13 minutes ago Up 13 minutes (healthy) 6379/tcp haf-world-redis-1
e02b34cae4ad registry.hive.blog/haf/minimal-instance:v1.27.5rc7 "/home/haf_admin/doc…" 13 minutes ago Up 13 minutes (healthy) 8091/tcp haf-world-haf-1
20fd71daf9ac registry.hive.blog/haf_api_node/version-display:latest "docker-entrypoint.s…" 13 minutes ago Up 13 minutes haf-world-version-display-1
eece23cdb395 registry.hive.blog/haf_api_node/caddy:latest "caddy run --config …" 13 minutes ago Up 13 minutes (healthy) 0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp, 0.0.0.0:443->443/udp, :::443->443/udp, 2019/tcp haf-world-caddy-1
1cfbe3c82a0f dpage/pgadmin4:8.3 "/entrypoint.sh" 13 minutes ago Up 13 minutes 80/tcp, 443/tcp haf-world-pgadmin-1
69320ef8c6ed registry.hive.blog/haf_api_node/haproxy-healthchecks:latest "/docker_entrypoint.…" 13 minutes ago Up 13 minutes haf-world-haproxy-healthchecks-1
6b2bb05d2786 haproxy:2.9.3-alpine "docker-entrypoint.s…" 13 minutes ago Up 13 minutes (healthy)
This is hivemind log:
Entrypoint] 2024-03-01 21:43:35,099+00:00 INFO [global] (main) Parameters passed directly to Hivemind docker entrypoint: sync --database-url=postgresql://hivemind@haf/haf_block_log --database-admin-url=postgresql://haf_admin@haf/haf_block_log --install-app
[Entrypoint] 2024-03-01 21:43:35,100+00:00 INFO [global] (main) Collected Hivemind arguments: sync
[Entrypoint] 2024-03-01 21:43:35,100+00:00 INFO [global] (main) Using PostgreSQL instance: postgresql://hivemind@haf/haf_block_log
[Entrypoint] 2024-03-01 21:43:35,101+00:00 INFO [global] (main) Using PostgreSQL Admin URL: postgresql://haf_admin@haf/haf_block_log
[Entrypoint] 2024-03-01 21:43:35,101+00:00 INFO [global] (main) Running install_app step because it was requested via the --install-app argument
[Entrypoint] 2024-03-01 21:43:35,102+00:00 INFO [setup] (main) Setting up the database...
./setup_postgres.sh parameters: --postgres-url=postgresql://haf_admin@haf/haf_block_log
/home/hivemind/app/../haf/scripts/create_haf_app_role.sh parameters: --postgres-url=postgresql://haf_admin@haf/haf_block_log --haf-app-account=hivemind
postgresql://haf_admin@haf/haf_block_log
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
DO $$
BEGIN
BEGIN
CREATE ROLE hivemind WITH LOGIN INHERIT IN ROLE hive_applications_owner_group;
EXCEPTION WHEN DUPLICATE_OBJECT THEN
RAISE NOTICE 'hivemind role already exists';
END;
END
$$;
DO
Attempting to supplement definition of hivemind builtin roles...
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
GRANT ROLE
./install_app.sh parameters: --postgres-url=postgresql://haf_admin@haf/haf_block_log
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
CREATE EXTENSION
psql:/home/hivemind/app/install_app.sql:19: NOTICE: Disabling a JIT optimization on the current database level...
DO
[Entrypoint] 2024-03-01 21:43:35,221+00:00 INFO [run_hive] (main) Starting Hivemind...
INFO - hive.conf:212 - The database instance is created...
INFO - hive.db.adapter:49 - A database offers maximum connections: 100. Required 15 connections.
INFO - hive.db.adapter:98 - Closing database connection: 'root'
INFO - hive.db.adapter:109 - Disposing SQL engine
INFO - hive.conf:261 - The database is disconnected...
INFO - hive.conf:212 - The database instance is created...
INFO - hive.db.db_state:48 - Welcome to hive!
INFO - hive.db.db_state:54 - Create db schema...
INFO - hive.indexer.hive_db.haf_functions:10 - Looking for 'hivemind_app' context.
INFO - hive.indexer.hive_db.haf_functions:13 - No application context present. Attempting to create a 'hivemind_app' context...
INFO - hive.indexer.hive_db.haf_functions:15 - Application context creation done.
INFO - hive.indexer.hive_db.haf_functions:31 - Trying to detach app context...
INFO - hive.indexer.hive_db.haf_functions:33 - App context detaching done.
WARNING - hive.db.adapter:276 - [SQL-ERR] IntegrityError in query CALL hive.appproc_context_attach('hivemind_app') ({})
INFO - hive.db.adapter:98 - Closing database connection: 'root'
INFO - hive.db.adapter:109 - Disposing SQL engine
INFO - hive.conf:261 - The database is disconnected...
Traceback (most recent call last):
File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
self.dialect.do_execute(
File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
cursor.execute(statement, parameters)
psycopg2.errors.NotNullViolation: null value in column "fork_id" of relation "contexts" violates not-null constraint
CONTEXT: SQL statement "UPDATE hive.contexts
SET fork_id = __fork_id
, irreversible_block = COALESCE( __head_of_irreversible_block, 0 )
, events_id = 0 -- during app_next_block correct event will be found
, last_active_at = NOW()
WHERE name =ANY( _contexts )"
PL/pgSQL function hive.app_context_attach(hive.contexts_group) line 29 at SQL statement
SQL statement "SELECT hive.app_context_attach( _contexts )"
PL/pgSQL function hive.appproc_context_attach(hive.contexts_group) line 3 at PERFORM
SQL statement "CALL hive.appproc_context_attach( ARRAY[ _context ] )"
PL/pgSQL function hive.appproc_context_attach(hive.context_name) line 3 at CALL
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/hivemind/.hivemind-venv/bin/hive", line 8, in <module>
sys.exit(run())
File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/hive/cli.py", line 65, in run
launch_mode(mode, conf)
File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/hive/cli.py", line 83, in launch_mode
with SyncHiveDb(conf=conf, enter_sync = False) as schema_builder:
File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/hive/indexer/sync.py", line 61, in __enter__
DbState.initialize(self._enter_sync)
File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/hive/db/db_state.py", line 57, in initialize
setup(admin_db=db_setup_admin, db=db_setup_owner)
File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/hive/db/schema.py", line 629, in setup
context_attach(db=db)
File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/hive/indexer/hive_db/haf_functions.py", line 46, in context_attach
db.query_no_return(f"CALL hive.appproc_context_attach('{SCHEMA_NAME}')")
File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/hive/db/adapter.py", line 167, in query_no_return
self._query(sql, **kwargs)
File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/hive/db/adapter.py", line 277, in _query
raise e
File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/hive/db/adapter.py", line 270, in _query
result = self._basic_connection.execution_options(autocommit=False).execute(query, **kwargs)
File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1385, in execute
return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection
return connection._execute_clauseelement(
File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1577, in _execute_clauseelement
ret = self._execute_context(
File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1953, in _execute_context
self._handle_dbapi_exception(
File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2134, in _handle_dbapi_exception
util.raise_(
File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
raise exception
File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
self.dialect.do_execute(
File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (psycopg2.errors.NotNullViolation) null value in column "fork_id" of relation "contexts" violates not-null constraint
CONTEXT: SQL statement "UPDATE hive.contexts
SET fork_id = __fork_id
, irreversible_block = COALESCE( __head_of_irreversible_block, 0 )
, events_id = 0 -- during app_next_block correct event will be found
, last_active_at = NOW()
WHERE name =ANY( _contexts )"
PL/pgSQL function hive.app_context_attach(hive.contexts_group) line 29 at SQL statement
SQL statement "SELECT hive.app_context_attach( _contexts )"
PL/pgSQL function hive.appproc_context_attach(hive.contexts_group) line 3 at PERFORM
SQL statement "CALL hive.appproc_context_attach( ARRAY[ _context ] )"
PL/pgSQL function hive.appproc_context_attach(hive.context_name) line 3 at CALL
[SQL: CALL hive.appproc_context_attach('hivemind_app')]
(Background on this error at: https://sqlalche.me/e/14/gkpj)
[Entrypoint] 2024-03-01 21:45:34,413+00:00 INFO [global] (main) Parameters passed directly to Hivemind docker entrypoint: sync --database-url=postgresql://hivemind@haf/haf_block_log --database-admin-url=postgresql://haf_admin@haf/haf_block_log --install-app
[Entrypoint] 2024-03-01 21:45:34,414+00:00 INFO [global] (main) Collected Hivemind arguments: sync
[Entrypoint] 2024-03-01 21:45:34,414+00:00 INFO [global] (main) Using PostgreSQL instance: postgresql://hivemind@haf/haf_block_log
[Entrypoint] 2024-03-01 21:45:34,415+00:00 INFO [global] (main) Using PostgreSQL Admin URL: postgresql://haf_admin@haf/haf_block_log
[Entrypoint] 2024-03-01 21:45:34,415+00:00 INFO [global] (main) Running install_app step because it was requested via the --install-app argument
[Entrypoint] 2024-03-01 21:45:34,416+00:00 INFO [setup] (main) Setting up the database...
./setup_postgres.sh parameters: --postgres-url=postgresql://haf_admin@haf/haf_block_log
/home/hivemind/app/../haf/scripts/create_haf_app_role.sh parameters: --postgres-url=postgresql://haf_admin@haf/haf_block_log --haf-app-account=hivemind
postgresql://haf_admin@haf/haf_block_log
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
DO $$
Attempting to supplement definition of hivemind builtin roles...
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
NOTICE: role "haf_admin" is already a member of role "hivemind"
GRANT ROLE
BEGIN
BEGIN
CREATE ROLE hivemind WITH LOGIN INHERIT IN ROLE hive_applications_owner_group;
EXCEPTION WHEN DUPLICATE_OBJECT THEN
RAISE NOTICE 'hivemind role already exists';
END;
END
$$;
psql:<stdin>:10: NOTICE: hivemind role already exists
DO
./install_app.sh parameters: --postgres-url=postgresql://haf_admin@haf/haf_block_log
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
psql:/home/hivemind/app/install_app.sql:2: NOTICE: extension "intarray" already exists, skipping
CREATE EXTENSION
psql:/home/hivemind/app/install_app.sql:19: NOTICE: Disabling a JIT optimization on the current database level...
DO
[Entrypoint] 2024-03-01 21:45:36,355+00:00 INFO [run_hive] (main) Starting Hivemind...
INFO - hive.conf:212 - The database instance is created...
INFO - hive.db.adapter:49 - A database offers maximum connections: 100. Required 15 connections.
INFO - hive.db.adapter:98 - Closing database connection: 'root'
INFO - hive.db.adapter:109 - Disposing SQL engine
INFO - hive.conf:261 - The database is disconnected...
INFO - hive.conf:212 - The database instance is created...
INFO - hive.db.db_state:48 - Welcome to hive!
INFO - hive.db.adapter:98 - Closing database connection: 'setup_owner'
INFO - hive.indexer.sync:67 - hivemind_version : 2.0.0dev1
INFO - hive.indexer.sync:68 - hivemind_git_rev : b50bd3dfe0a12b07f1fcfe8b2817217d83e38bae
INFO - hive.indexer.sync:69 - hivemind_git_date : 2024-02-27 21:44:25
INFO - hive.indexer.sync:71 - database_schema_version : 34
INFO - hive.indexer.sync:72 - database_patch_date : 2024-03-01 21:45:47.019338
INFO - hive.indexer.sync:73 - database_patched_to_revision : 9d2cc15bea71a39139abdf49569e0eac6dd0b970
INFO - hive.indexer.sync:75 - last_block_from_view : 0
INFO - hive.indexer.sync:76 - last_imported_block : 1
INFO - hive.indexer.sync:77 - last_completed_block : 1
INFO - hive.indexer.sync:92 - Attempting to build Hivemind database schema if needed
INFO - hive.db.adapter:98 - Closing database connection: 'root'
INFO - hive.db.adapter:109 - Disposing SQL engine
INFO - hive.conf:261 - The database is disconnected...
[Entrypoint] 2024-03-01 21:45:47,108+00:00 INFO [global] (main) Done running install_app, now running the block processor
[Entrypoint] 2024-03-01 21:45:47,109+00:00 INFO [run_hive] (main) Starting Hivemind...
INFO - hive.conf:212 - The database instance is created...
INFO - hive.db.adapter:49 - A database offers maximum connections: 100. Required 15 connections.
INFO - hive.db.adapter:98 - Closing database connection: 'root'
INFO - hive.db.adapter:109 - Disposing SQL engine
INFO - hive.conf:261 - The database is disconnected...
INFO - hive.conf:212 - The database instance is created...
INFO - hive.indexer.sync:54 - Entering HAF mode synchronization
INFO - hive.db.db_state:48 - Welcome to hive!
INFO - hive.db.adapter:98 - Closing database connection: 'setup_owner'
INFO - hive.db.db_state:67 - [MASSIVE] Continue with massive sync...
INFO - hive.indexer.sync:67 - hivemind_version : 2.0.0dev1
INFO - hive.indexer.sync:68 - hivemind_git_rev : b50bd3dfe0a12b07f1fcfe8b2817217d83e38bae
INFO - hive.indexer.sync:69 - hivemind_git_date : 2024-02-27 21:44:25
INFO - hive.indexer.sync:71 - database_schema_version : 34
INFO - hive.indexer.sync:72 - database_patch_date : 2024-03-01 21:45:47.019338
INFO - hive.indexer.sync:73 - database_patched_to_revision : 9d2cc15bea71a39139abdf49569e0eac6dd0b970
INFO - hive.indexer.sync:75 - last_block_from_view : 0
INFO - hive.indexer.sync:76 - last_imported_block : 1
INFO - hive.indexer.sync:77 - last_completed_block : 1
WARNING - hive.db.adapter:276 - [SQL-ERR] IntegrityError in query CALL hive.appproc_context_attach('hivemind_app') ({})
INFO - hive.db.adapter:98 - Closing database connection: 'root'
INFO - hive.db.adapter:109 - Disposing SQL engine
INFO - hive.conf:261 - The database is disconnected...
Traceback (most recent call last):
File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
self.dialect.do_execute(
File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
cursor.execute(statement, parameters)
psycopg2.errors.NotNullViolation: null value in column "fork_id" of relation "contexts" violates not-null constraint
CONTEXT: SQL statement "UPDATE hive.contexts
SET fork_id = __fork_id
, irreversible_block = COALESCE( __head_of_irreversible_block, 0 )
, events_id = 0 -- during app_next_block correct event will be found
, last_active_at = NOW()
WHERE name =ANY( _contexts )"
PL/pgSQL function hive.app_context_attach(hive.contexts_group) line 29 at SQL statement
SQL statement "SELECT hive.app_context_attach( _contexts )"
PL/pgSQL function hive.appproc_context_attach(hive.contexts_group) line 3 at PERFORM
SQL statement "CALL hive.appproc_context_attach( ARRAY[ _context ] )"
PL/pgSQL function hive.appproc_context_attach(hive.context_name) line 3 at CALL
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/hivemind/.hivemind-venv/bin/hive", line 8, in <module>
sys.exit(run())
File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/hive/cli.py", line 65, in run
launch_mode(mode, conf)
File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/hive/cli.py", line 94, in launch_mode
with SyncHiveDb(conf=conf, enter_sync = True) as sync:
File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/hive/indexer/sync.py", line 68, in __enter__
context_attach(db=self._db)
File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/hive/indexer/hive_db/haf_functions.py", line 46, in context_attach
db.query_no_return(f"CALL hive.appproc_context_attach('{SCHEMA_NAME}')")
File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/hive/db/adapter.py", line 167, in query_no_return
self._query(sql, **kwargs)
File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/hive/db/adapter.py", line 277, in _query
raise e
File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/hive/db/adapter.py", line 270, in _query
result = self._basic_connection.execution_options(autocommit=False).execute(query, **kwargs)
File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1385, in execute
return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection
return connection._execute_clauseelement(
File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1577, in _execute_clauseelement
ret = self._execute_context(
File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1953, in _execute_context
self._handle_dbapi_exception(
File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2134, in _handle_dbapi_exception
util.raise_(
File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
raise exception
File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
self.dialect.do_execute(
File "/home/hivemind/.hivemind-venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (psycopg2.errors.NotNullViolation) null value in column "fork_id" of relation "contexts" violates not-null constraint
CONTEXT: SQL statement "UPDATE hive.contexts
SET fork_id = __fork_id
, irreversible_block = COALESCE( __head_of_irreversible_block, 0 )
, events_id = 0 -- during app_next_block correct event will be found
, last_active_at = NOW()
WHERE name =ANY( _contexts )"
PL/pgSQL function hive.app_context_attach(hive.contexts_group) line 29 at SQL statement
SQL statement "SELECT hive.app_context_attach( _contexts )"
PL/pgSQL function hive.appproc_context_attach(hive.contexts_group) line 3 at PERFORM
SQL statement "CALL hive.appproc_context_attach( ARRAY[ _context ] )"
PL/pgSQL function hive.appproc_context_attach(hive.context_name) line 3 at CALL
[SQL: CALL hive.appproc_context_attach('hivemind_app')]
(Background on this error at: https://sqlalche.me/e/14/gkpj)