Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[future][discussion] deal with data located in files #196

Open
mathysgrapotte opened this issue Oct 23, 2024 · 0 comments
Open

[future][discussion] deal with data located in files #196

mathysgrapotte opened this issue Oct 23, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@mathysgrapotte
Copy link
Contributor

Description of feature

Current implementation requires a flat-file input csv file where one line = one entry (see readme ).

However, most datasets are not flat, i.e. image datasets require data to be saved in external files

We should consider :

  1. the shuffle process shuffles the lines of the input csv file, image datasets have few files that each contain lots of information (which would decrease the effectiveness of shuffle). idem for the split method.
  2. how to mount those files to the various processes ?
  3. effective way of parsing those files
  4. memory allocation for large datasets
@mathysgrapotte mathysgrapotte added the enhancement New feature or request label Oct 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: Todo
Development

No branches or pull requests

1 participant