Skip to content

Fix slowdown on cache update

Michal Zander requested to merge mzander/fix-cache into develop

Commited changes in this MR are first attepmt do fix the issue, ive tried to aggregate results in hive.account_operations, but the current indexes in this table lacks (previously existing) index on (op_type_id, account_id) - now exists only (account_id, op_type_id) - that doesn't allow such aggregation and this commited query is slower than develop

Another try:

explain analyze
  SELECT 
  cw.witness_id, 
  COALESCE(
  (
	  
-----------------------------INTRESTING PART-------------------------------------------
-- (1) first i find the newest update
    WITH select_witness_update as MATERIALIZED
    (
      SELECT MAX(aov.operation_id) as operation_id
      FROM hive.account_operations_view aov 
      WHERE aov.op_type_id = 11 and aov.account_id = cw.witness_id
    ),
	join_witness_update AS MATERIALIZED
	(
	  SELECT cw.witness_id as account_id, swu.operation_id, ov.body_binary::JSONB as body
	  FROM select_witness_update swu
	  JOIN hive.operations_view ov ON ov.id = swu.operation_id
	),
	select_all_witness_set_properties_ops AS MATERIALIZED 
    (
-- (2) i look if AFTER the update happened any witness_set_property
      SELECT aov.account_id, aov.operation_id
      FROM hive.account_operations_view aov
      WHERE aov.op_type_id = 42 AND aov.operation_id > (SELECT operation_id FROM select_witness_update)
      AND aov.account_id = cw.witness_id
	),
-- (3) IF it did operations are being filtered by `join_operations` (this part is killing the query aswell)
	join_operations as MATERIALIZED
	(
	SELECT saw.account_id, saw.operation_id, 
		(SELECT ((extract_set_witness_properties.prop_value)::TEXT)::INT FROM hive.extract_set_witness_properties(
          json_build_array(
            json_build_array(
              'hbd_interest_rate',
              (ov.body->'value'->'props'->0->>1)::text
            )
          )::TEXT 
        )) AS hbd_interest_rate
	FROM select_all_witness_set_properties_ops saw
	JOIN hive.operations_view ov ON ov.id = saw.operation_id
	WHERE (ov.body->'value'->'props'->0->>0)::TEXT = 'hbd_interest_rate'
	ORDER BY saw.operation_id DESC
	LIMIT 1
	)
--(4) if `join_operations` is not empty then the hbd_interest_rate is taken from there, if it is emty that means operation found in (1) step has final value of hbd_interest_rate
  	SELECT 
	  (CASE WHEN ja.operation_id IS NULL THEN
	  (jwu.body->'value'->'props'->>'hbd_interest_rate')::INT
	  ELSE 
	  ja.hbd_interest_rate
	  END
	  ) AS hbd_interest_rate
	FROM join_witness_update jwu
	LEFT JOIN join_operations ja ON ja.account_id = jwu.account_id
------------------------------------------------------------------------
  )
    ,0) as hbd_interest_rate,
  COALESCE(
  (
    SELECT count(*) as missed
    FROM hive.account_operations_view aov 
    WHERE aov.op_type_id = 86 AND aov.account_id = cw.witness_id
  )
  ,0) as missed_blocks
  FROM hafbe_app.current_witnesses cw 

Above query is killed by the filtering part too

Another try:

This query works fine - very similar to develop without 42 operation (tested on steem16 and steem13) but the result is lacking the possibility of change the property by 42 operation in the future (curretly there is 1 witness out of 1355 active, that has his last update of hbd_interest_rate was made by 42) - so the query result has his property wrong.

  explain analyze
  SELECT 
  cw.witness_id, 
  COALESCE(
  (
    WITH select_witness_update as MATERIALIZED
    (
      SELECT  MAX(aov.operation_id) as top_op_id
      FROM hive.account_operations_view aov 
      WHERE aov.op_type_id = 11 and aov.account_id = cw.witness_id
    )
    SELECT (body_binary::JSONB->'value'->'props'->>'hbd_interest_rate')::INT as hbd_interest_rate
    FROM hive.operations_view ov 
    JOIN select_witness_update swu ON ov.id = swu.top_op_id
  )
    ,0) as hbd_interest_rate,
  COALESCE(
  (
    SELECT count(*) as missed
    FROM hive.account_operations_view aov 
    WHERE aov.op_type_id = 86 AND aov.account_id = cw.witness_id
  )
  ,0) as missed_blocks
  FROM hafbe_app.current_witnesses cw 

As i metioned the above query lacks the possiblity of witness_set_property in the future, so i think this can be resolved by:

  1. adding index for 42 operation and adjusting above query (index pointing to props body_binary::JSONB->'value'->'props'->0->>0)
  2. calculating it per block_range and updating new table in hafbe_sync
Edited by Michal Zander

Merge request reports