Further reaserch related to table partitions
Some initial reasearch has been done here: !360 Probably selected approach was wrong and too much partitions has been created. Anyway idea to use partitions can be useful to remove explicit reversible tables and store HAF data as single logical tables splitted as follows:
- reversible part - determined by some column value (fork_id - which has set nonzero value in this case) best being stored as part of block_num value
- hot irreversible part - determined by fork_id = 0 and block_num and storing fresh N blocks (i.e. last 5M what covers c.a. half of year)
- archive irreversible part - for all older block_nums.
Benefits:
- simplification of complex definition of data access views (where right now UNION ALL statements must be specified
- speedup of data write mostly in LIVE sync. In test done , multiple archive partitions also shortened overall replay time, although to high number can lead to API query degradation (in test selected threshold was 1M what was too small).
- makes possible split data across separate storages having different speed (and also cost)