During the initialization of reader_2stage, metadata from all input files are currently inspected sequentially by rank 0. For datasets containing a large number of files, this can result in significant initialization time.
I propose distributing the metadata inspection across the available reader ranks, gathering the resulting metadata on rank 0, and restoring the original file ordering before constructing _file_time, _shape, and related attributes.
This does not modify the data-reading algorithm itself, only the initialization step.
During the initialization of reader_2stage, metadata from all input files are currently inspected sequentially by rank 0. For datasets containing a large number of files, this can result in significant initialization time.
I propose distributing the metadata inspection across the available reader ranks, gathering the resulting metadata on rank 0, and restoring the original file ordering before constructing _file_time, _shape, and related attributes.
This does not modify the data-reading algorithm itself, only the initialization step.