hafbe tries to write duplicate data into sync_time_logs

Got this log after shutting down a node, then bringing it back up after a few days. This is the complete log:

2023-11-15T21:59:58.469157727Z Timing is on.
2023-11-15T21:59:58.478267883Z NOTICE:  Last block processed by application: 80135935
2023-11-15T21:59:58.480622607Z NOTICE:  Entering application main loop...
2023-11-15T21:59:58.482199906Z NOTICE:  HAF instance is ready. Exiting wait loop...
2023-11-15T21:59:58.505552612Z NOTICE:  Attempting to process block range: <80135936,80135936>
2023-11-15T21:59:58.505766910Z NOTICE:  Starting single block processing: 80135936
2023-11-15T21:59:58.540704415Z NOTICE:  btracker_app_a processed successfully. 0.034 seconds
2023-11-15T21:59:58.561688117Z NOTICE:  Updated last processed block.
2023-11-15T21:59:58.561722167Z NOTICE:  btracker_app_b processed successfully. 0.021 seconds
2023-11-15T21:59:58.564814126Z NOTICE:  hafbe_app_a processed successfully. 0.003 seconds
2023-11-15T21:59:58.570289127Z NOTICE:  hafbe_app_b processed successfully. 0.005 seconds
2023-11-15T21:59:58.591501318Z NOTICE:  hafbe_app_c processed successfully. 0.021 seconds
2023-11-15T21:59:58.595190832Z NOTICE:  state_provider processed successfully. 0.004 seconds
2023-11-15T21:59:58.598366230Z NOTICE:  Process witness cache...
2023-11-15T21:59:58.598466839Z NOTICE:  Updating witnesses caches
2023-11-15T22:00:13.151592219Z NOTICE:  Updated witness voters cache
2023-11-15T22:00:15.283937979Z NOTICE:  Updated witness voters history cache
2023-11-15T22:00:15.306133273Z NOTICE:  Updated witnesses cache
2023-11-15T22:00:15.381890670Z NOTICE:  Updated witness change cache
2023-11-15T22:00:15.382080058Z NOTICE:  Witness cache processing done.
2023-11-15T22:00:15.382656874Z NOTICE:  HAF instance is ready. Exiting wait loop...
2023-11-15T22:00:15.563102265Z NOTICE:  Attempting to process block range: <80135937,80135937>
2023-11-15T22:00:15.563120275Z NOTICE:  Starting single block processing: 80135937
2023-11-15T22:00:15.570229004Z NOTICE:  btracker_app_a processed successfully. 0.007 seconds
2023-11-15T22:00:15.575188500Z NOTICE:  Updated last processed block.
2023-11-15T22:00:15.575210790Z NOTICE:  btracker_app_b processed successfully. 0.005 seconds
2023-11-15T22:00:15.575745626Z NOTICE:  hafbe_app_a processed successfully. 0.000 seconds
2023-11-15T22:00:15.577226686Z NOTICE:  hafbe_app_b processed successfully. 0.001 seconds
2023-11-15T22:00:15.584578864Z NOTICE:  hafbe_app_c processed successfully. 0.007 seconds
2023-11-15T22:00:15.587126086Z NOTICE:  state_provider processed successfully. 0.002 seconds
2023-11-15T22:00:15.590395933Z NOTICE:  HAF instance is ready. Exiting wait loop...
2023-11-15T22:00:15.611735873Z NOTICE:  Attempting to process block range: <80135938,80135938>
2023-11-15T22:00:15.611766783Z NOTICE:  Starting single block processing: 80135938
2023-11-15T22:00:15.617889129Z NOTICE:  btracker_app_a processed successfully. 0.006 seconds
2023-11-15T22:00:15.620619590Z NOTICE:  Updated last processed block.
2023-11-15T22:00:15.620660220Z NOTICE:  btracker_app_b processed successfully. 0.003 seconds
2023-11-15T22:00:15.621157306Z NOTICE:  hafbe_app_a processed successfully. 0.000 seconds
2023-11-15T22:00:15.622624746Z NOTICE:  hafbe_app_b processed successfully. 0.001 seconds
2023-11-15T22:00:15.632263418Z NOTICE:  hafbe_app_c processed successfully. 0.010 seconds
2023-11-15T22:00:15.634092006Z NOTICE:  state_provider processed successfully. 0.002 seconds
2023-11-15T22:00:15.637324712Z NOTICE:  HAF instance is ready. Exiting wait loop...
2023-11-15T22:00:15.657792349Z NOTICE:  Attempting to process block range: <80135939,80135939>
2023-11-15T22:00:15.657808648Z NOTICE:  Starting single block processing: 80135939
2023-11-15T22:00:15.661589272Z NOTICE:  btracker_app_a processed successfully. 0.004 seconds
2023-11-15T22:00:15.714198202Z NOTICE:  Updated last processed block.
2023-11-15T22:00:15.714218012Z NOTICE:  btracker_app_b processed successfully. 0.053 seconds
2023-11-15T22:00:15.714878517Z NOTICE:  hafbe_app_a processed successfully. 0.001 seconds
2023-11-15T22:00:15.716675414Z NOTICE:  hafbe_app_b processed successfully. 0.002 seconds
2023-11-15T22:00:15.722787391Z NOTICE:  hafbe_app_c processed successfully. 0.006 seconds
2023-11-15T22:00:15.724664588Z NOTICE:  state_provider processed successfully. 0.002 seconds
2023-11-15T22:00:15.728157123Z NOTICE:  HAF instance is ready. Exiting wait loop...
2023-11-15T22:00:15.748960457Z NOTICE:  Attempting to process block range: <80135940,80135940>
2023-11-15T22:00:15.748970087Z NOTICE:  Starting single block processing: 80135940
2023-11-15T22:00:15.750403397Z NOTICE:  btracker_app_a processed successfully. 0.001 seconds
2023-11-15T22:00:15.751395460Z NOTICE:  Updated last processed block.
2023-11-15T22:00:15.751423480Z NOTICE:  btracker_app_b processed successfully. 0.001 seconds
2023-11-15T22:00:15.751928036Z NOTICE:  hafbe_app_a processed successfully. 0.000 seconds
2023-11-15T22:00:20.922683317Z NOTICE:  hafbe_app_b processed successfully. 5.171 seconds
2023-11-15T22:00:20.931712853Z NOTICE:  hafbe_app_c processed successfully. 0.009 seconds
2023-11-15T22:00:20.933520970Z NOTICE:  state_provider processed successfully. 0.002 seconds
2023-11-15T22:00:20.936678309Z NOTICE:  HAF instance is ready. Exiting wait loop...
2023-11-15T22:00:20.958046138Z NOTICE:  Attempting to process block range: <80135941,80135941>
2023-11-15T22:00:20.958067838Z NOTICE:  Starting single block processing: 80135941
2023-11-15T22:00:20.961852921Z NOTICE:  btracker_app_a processed successfully. 0.004 seconds
2023-11-15T22:00:20.968593194Z NOTICE:  Updated last processed block.
2023-11-15T22:00:20.968609143Z NOTICE:  btracker_app_b processed successfully. 0.007 seconds
2023-11-15T22:00:20.969129440Z NOTICE:  hafbe_app_a processed successfully. 0.000 seconds
2023-11-15T22:00:20.970510320Z NOTICE:  hafbe_app_b processed successfully. 0.001 seconds
2023-11-15T22:00:20.981618152Z NOTICE:  hafbe_app_c processed successfully. 0.011 seconds
2023-11-15T22:00:20.984240244Z NOTICE:  state_provider processed successfully. 0.003 seconds
2023-11-15T22:00:20.987953618Z NOTICE:  HAF instance is ready. Exiting wait loop...
2023-11-15T22:00:21.008925710Z NOTICE:  Attempting to process block range: <80135942,80135942>
2023-11-15T22:00:21.008941880Z NOTICE:  Starting single block processing: 80135942
2023-11-15T22:00:21.013684307Z NOTICE:  btracker_app_a processed successfully. 0.005 seconds
2023-11-15T22:00:21.013976125Z NOTICE:  Updated last processed block.
2023-11-15T22:00:21.013998024Z NOTICE:  btracker_app_b processed successfully. 0.000 seconds
2023-11-15T22:00:21.014131504Z NOTICE:  hafbe_app_a processed successfully. 0.000 seconds
2023-11-15T22:00:21.014434311Z NOTICE:  hafbe_app_b processed successfully. 0.000 seconds
2023-11-15T22:00:21.019767964Z NOTICE:  hafbe_app_c processed successfully. 0.005 seconds
2023-11-15T22:00:21.021347373Z NOTICE:  state_provider processed successfully. 0.002 seconds
2023-11-15T22:00:21.024674299Z NOTICE:  HAF instance is ready. Exiting wait loop...
2023-11-15T22:00:21.081314920Z WARNING:  Waiting for next block...
2023-11-15T22:00:21.081583048Z NOTICE:  HAF instance is ready. Exiting wait loop...
2023-11-15T22:00:21.104801025Z WARNING:  Waiting for next block...
2023-11-15T22:00:21.105075003Z NOTICE:  HAF instance is ready. Exiting wait loop...
2023-11-15T22:00:21.131075290Z WARNING:  Waiting for next block...
2023-11-15T22:00:21.131294178Z NOTICE:  HAF instance is ready. Exiting wait loop...
2023-11-15T22:00:21.154578695Z WARNING:  Waiting for next block...
2023-11-15T22:00:21.154827463Z NOTICE:  HAF instance is ready. Exiting wait loop...
2023-11-15T22:00:21.177876020Z WARNING:  Waiting for next block...
2023-11-15T22:00:21.178130329Z NOTICE:  HAF instance is ready. Exiting wait loop...
2023-11-15T22:00:21.201153297Z WARNING:  Waiting for next block...
2023-11-15T22:00:21.201434495Z NOTICE:  HAF instance is ready. Exiting wait loop...
2023-11-15T22:00:21.224576432Z WARNING:  Waiting for next block...
2023-11-15T22:00:21.224848250Z NOTICE:  HAF instance is ready. Exiting wait loop...
2023-11-15T22:00:21.269197058Z WARNING:  Waiting for next block...
2023-11-15T22:00:21.269460296Z NOTICE:  HAF instance is ready. Exiting wait loop...
2023-11-15T22:00:21.309750812Z NOTICE:  Attempting to process block range: <80135940,80194243>
2023-11-15T22:00:21.309872371Z NOTICE:  Entering massive processing of block range: <80135940, 80194243>...
2023-11-15T22:00:21.309881511Z NOTICE:  Detaching HAF application context...
2023-11-15T22:00:21.330654525Z NOTICE:  Attempting to process a block range: <80135940, 80145939>
2023-11-15T22:00:26.895623949Z NOTICE:  btracker_app_a processed successfully. 5.565 seconds
2023-11-15T22:00:36.940735863Z NOTICE:  Updated last processed block.
2023-11-15T22:00:36.940800152Z NOTICE:  btracker_app_b processed successfully. 10.045 seconds
2023-11-15T22:00:36.960739681Z NOTICE:  hafbe_app_a processed successfully. 0.020 seconds
2023-11-15T22:00:37.221275325Z NOTICE:  hafbe_app_b processed successfully. 0.260 seconds
2023-11-15T22:00:43.883787166Z NOTICE:  hafbe_app_c processed successfully. 6.662 seconds
2023-11-15T22:00:43.904269532Z NOTICE:  state_provider processed successfully. 0.020 seconds
2023-11-15T22:00:43.904657559Z ERROR:  duplicate key value violates unique constraint "pk_massive_sync_time_logs"
2023-11-15T22:00:43.904666519Z DETAIL:  Key (block_num)=(80135940) already exists.
2023-11-15T22:00:43.904670019Z CONTEXT:  SQL statement "INSERT INTO hafbe_app.sync_time_logs VALUES (_from, _time)"
2023-11-15T22:00:43.904672909Z PL/pgSQL function hafbe_app.massive_processing_step(character varying,integer,integer) line 33 at SQL statement
2023-11-15T22:00:43.904676079Z SQL statement "CALL hafbe_app.massive_processing_step(_appContext, b, _last_block)"
2023-11-15T22:00:43.904678979Z PL/pgSQL function hafbe_app.do_massive_processing(character varying,character varying,integer,integer,integer,integer) line 15 at CALL
2023-11-15T22:00:43.904681659Z SQL statement "CALL hafbe_app.do_massive_processing(_appContext, _appContext_btracker, __next_block_range.first_block, __next_block_range.last_block, 10000, __last_block)"
2023-11-15T22:00:43.904684469Z PL/pgSQL function hafbe_app.main(character varying,character varying,integer) line 47 at CALL
2023-11-15T22:00:43.904736678Z Time: 45435.504 ms (00:45.436)

Looking at the sync_time_logs table after the error, it has entries for blocks 80135930 - 80135942 (and lots of blocks before that too, but they didn't seem relevant)