Previously sync cannot leave MASSIVE_WITH_INDEXES because update_all_hive_posts_children_count()
and VACUUM were called

when new sync loop was introduced then information about last block in
massive was lost, and instead end block of each batch was set, what
make unable to ignore outdated effective comment vote  operations

it works now as it worked before introducing new synchronization loop

Previous query was faster on postgres server site, but its results was much slower processed on python site

From now during massive sync blocks are taken from HAF in the main thread, then processed in another thread. The main loop do not wait to finish processing  a batch of blocks, but backs to getting new blocks while processing is in progress.

- update HAF to version with a new application loop
- modify hivemind sync loop:
	- define stages for hivemind context:
	 MASSIVE_WITHOUT_INDEXES, MASSIVE_WITH_INDEXES, live
	- stop retrieving block data in a separate thread. The queues for block data were consistently full,
	  indicating that querying for blocks is negligible compared to processing them.
	  Note: This change may impact sync performance.
	- Stop recalculating reputation during sync transition from MASSIVE_WITHOUT_INDEXES to MASSIVE_WITH_INDEXES.
	  Reputation will now be recalculated when entering the 'live' state. Note: This change may impact sync performance.

exec is required to correctly handle signals by hivemind
python code, without exec (when there is no additional processes between
hivemind and linux)  signals are not delivered to hivemind

exec was removed wit commit: 1751bc56

Utility functions raising exceptions have specified log_min_messages=FATAL to eliminate exception messages from SQL server logs

Implementation wrapper functions should not use `return_error_info` decorator to avoid log pollution by propagated error messages.

Bartek Wrona authored 1 year ago and Dan Notestein committed 1 year ago

Upgrade code should be executed conditionally to avoid errors when upgrade is not applicable to given version and should be skipped.

Bartek Wrona authored 1 year ago and Dan Notestein committed 1 year ago

One of hive_follows indexes extended by `id` column to eliminate HEAP sort while calling condenser_get_followers

Fixed upgrade_table_schema/upgrade_runtime_migration bugs related to hive_post_tags table data migration.

Removed creation of hive_posts (tags_ids) GIN indexes from upgrade code, due to new tags implementation