Skip to content

Parallelize file metadata inspection in 2-stage reader - #75

Open
Baptiste-Arnould wants to merge 1 commit into
MathEXLab:mainfrom
Baptiste-Arnould:feature/parallel-metadata-inspection
Open

Parallelize file metadata inspection in 2-stage reader#75
Baptiste-Arnould wants to merge 1 commit into
MathEXLab:mainfrom
Baptiste-Arnould:feature/parallel-metadata-inspection

Conversation

@Baptiste-Arnould

Copy link
Copy Markdown

Summary

This PR parallelizes file metadata inspection during the initialization of the 2-stage reader.

Previously, metadata for all input files were inspected sequentially by rank 0. For datasets containing a large number of files, this can result in significant initialization time.

The metadata inspection is now distributed across the available reader ranks. The resulting metadata are gathered on rank 0 and reordered according to the original file order before constructing _file_time, _shape, _is_real, and _files_size.

The data-reading algorithm itself is unchanged.

Changes

  • Add a helper function to inspect file metadata.
  • Distribute metadata inspection across reader ranks.
  • Gather metadata on rank 0.
  • Restore the original file ordering before constructing reader metadata.

Testing

Tested with MPI on a dataset composed of 11800 files (~250TB in total).

Closes #74

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Parallelize file metadata inspection in 2-stage reader

1 participant