Skip to content

Use memory mapped file io to speedup rebuilding block index file.

Dan Notestein requested to merge dn_fast_index_rebuild into master

The old code for constructing the block index from the block log file used a forward-walking strategy to build the block log and took about 15 minutes on a high speed SSD to build the block log from scratch. This new one using a combination of a backwards-walking strategy and memory-mapped IO is over 3x faster. There's likely to be further gains when using magnetic drives.

This uses unix-only mmap system call. If this code lasts for a while and we later want to support windows, we should consider replacing with a platform-independent class for it.

Merge request reports