hive sync got stuck at block 4999980 after SIGINT
I was running hivemind sync in a loop sending SIGINT and restarting sync. Near the end of massive sync the hivemind process got stuck at
INFO - hive.indexer.sync:242 - Next block range from hive.app_next_iteration is: <4999980:4999980>Logs:
haf_block_log> SELECT hive.get_current_stage_name('hivemind_app')
+-------------------------+
| get_current_stage_name |
|-------------------------|
| MASSIVE_WITHOUT_INDEXES |
+-------------------------+haf_block_log=# table contexts;
id | name | schema | current_block_num | irreversible_block | back_from_fork | events_id | fork_id | owner | registering_state_provider | is_forking | last_active_at | baseclass_id | stages | loop
----+----------------+----------------+-------------------+--------------------+----------------+---------------------+---------+------------------+----------------------------+------------+----------------------------+-------------------------------+---------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------
2 | hivemind_app | hivemind_app | 4999979 | 5000024 | f | 9223372036854775807 | 2 | hivemind | f | f | 2025-07-08 11:36:23.205034 | hivemind_app.hivemind_app | {"(MASSIVE_WITHOUT_INDEXES,201600,1000,00:00:20)","(MASSIVE_WITH_INDEXES,101,1000,00:00:20)","(live,0,1,00:00:03)"} | (5000024,"(MASSIVE_WITHOUT_INDEXES,201600,1000,00:00:20)",4999979,1000,4999979)
1 | reptracker_app | reptracker_app | 4999979 | 5000024 | f | 9223372036854775807 | 2 | reptracker_owner | f | t | 2025-07-08 08:36:57.813833 | reptracker_app.reptracker_app | {"(MASSIVE_PROCESSING,101,10000,00:00:10)","(live,0,1,00:00:03)"} | (5000024,"(MASSIVE_PROCESSING,101,10000,00:00:10)",4999979,10000,4999979)
(2 rows)There's a blocked connection in pg_top:
PID USERNAME SIZE RES STATE XTIME QTIME %CPU LOCKS COMMAND
194705 hivemind 230M 40M active 71:32 71:32 0.0 37 ? INSERT INTO hivemind_app.hive_accounts (name, created_at, posting_json_metadata, json_metadata, haf_id )? VALUESThe problematic query:
haf_block_log=# SELECT pid, state, query_start, query FROM pg_stat_activity WHERE pid = 194671 OR query LIKE '%hive%' ORDER BY query_start;
pid | state | query_start | query
--------+---------------------+-------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------
194673 | idle in transaction | 2025-07-08 11:36:23.493861+00 | SELECT hive.get_current_stage_name('hivemind_app')
194706 | idle | 2025-07-08 11:36:23.495688+00 | +
| | | SELECT hpd.body FROM hivemind_app.hive_post_data hpd WHERE hpd.id = 1257490; +
| | |
194705 | active | 2025-07-08 11:36:23.499969+00 | +
| | | INSERT INTO hivemind_app.hive_accounts (name, created_at, posting_json_metadata, json_metadata, haf_id ) +
| | | VALUES ( 'sasharaven', '2016-09-15T18:55:45', '', '', (select ha.id from hivemind_app.accounts_view ha where ha.name = 'sasharaven') )+
| | | RETURNING id +
| | |
40974 | idle | 2025-07-08 13:03:25.155701+00 | SELECT pid, state, query_start, query FROM pg_stat_activity WHERE pid = 194671 OR query LIKE '%hive%' ORDER BY query_start
205522 | active | 2025-07-08 13:04:15.443898+00 | SELECT pid, state, query_start, query FROM pg_stat_activity WHERE pid = 194671 OR query LIKE '%hive%' ORDER BY query_start
(5 rows)Claude Code diagnosis
The Python process (PID 194671) is stuck in a graceful shutdown sequence after receiving a SIGINT signal. Based on the analysis:
Current State:
- Process is waiting on a futex (userspace mutex) at hive/indexer/sync.py:197
- The _wait_for_massive_consume() method is calling self._massive_consume_blocks_futures.result()
- This is blocking waiting for a ThreadPoolExecutor task to complete
Root Cause:
The process received SIGINT and entered graceful shutdown mode, but the background thread handling massive block consumption is not responding to the shutdown signal. The main thread is stuck waiting at
sync.py:197 for the massive consume thread to finish.
Why it's stuck:
- SIGINT triggered the signal handler which set FINISH_SIGNAL_DURING_SYNC = 1
- Main sync loop detected the signal via can_continue_thread() returning False
- Code called _wait_for_massive_consume() to wait for background thread cleanup
- The background thread is likely stuck in a database operation or similar blocking call
- The ThreadPoolExecutor.result() call is waiting indefinitely for the thread to complete
Recommendations:
- Force kill: kill -9 194671 to terminate immediately
- Check database: Background thread may be waiting on a database lock/transaction
- Add timeout: The _wait_for_massive_consume() method should use a timeout on .result()
- Improve signal handling: Background threads should check can_continue_thread() periodically
Cursor agent diagnosis
Root Cause: Database Deadlock
The Python process 194671 is waiting for the database operation to complete, but it's stuck because of the deadlock.
Why This Happened
Looking at the code in hive/indexer/sync.py, the issue occurs in the _query_for_app_next_block() method:
def _query_for_app_next_block(self) -> Tuple[int, int]:
# ...
result = self._db.query_one("CALL hive.app_next_iteration(...)")
self._db._trx_active=True # This sets transaction as active
# ...The process calls hive.app_next_iteration() which starts a transaction, but then gets stuck waiting for the next operation. The transaction remains open ("idle in transaction") and blocks other operations.
Current State
- The process is in MASSIVE_WITHOUT_INDEXES mode
- It has processed blocks up to 4999979
- It's trying to get the next block range but is blocked by the database deadlock
Recommended Solutions
- Immediate Fix: Kill the stuck database connections
- Restart the hivemind process
- Long-term Fix: The code should be modified to ensure proper transaction handling in the _query_for_app_next_block() method to prevent long-running idle transactions. The process should resume normally after clearing the database deadlock.