* implement SourceUnitEnumerator and SourceUnitChunker
* natively support zstd-compressed inputs
* allow it to be used in multi-source config
Previously, multiple files would be processed sequentially, and zstd-compressed files would have to be decompressed externally.
These changes allow for slightly more efficient scanning of zstd-compressed NDJSON inputs (less CPU required), and enable greater CPU utilization on multicore systems when scanning multiple NDJSON files. With these changes, I see a 1.18x speedup in wall clock time in a simple experiment of scanning 10 zstd-compressed NDJSON files (~9GiB on disk) with verification disabled.
Brad Larsen
f5370819b5