Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Auto Download #169

Open
GaoxiangLuo opened this issue Jul 6, 2022 · 0 comments
Open

[Feature] Auto Download #169

GaoxiangLuo opened this issue Jul 6, 2022 · 0 comments
Labels
enhancement New feature or request

Comments

@GaoxiangLuo
Copy link
Collaborator

Is your feature request related to a problem? Please describe.
Some existing examples' datasets come from the installed python packages, but when it comes to users' own datasets, downloading the data is a necessary step. It's good to have the data download method integrated into the mode folder as part of the flame python library. Meanwhile, the design of flame system recognizes which dataset can be accessed by which cluster node within the firewall. It maybe good to take this aspect into consideration.

Describe the solution you'd like
As users' data can be stored at various places (e.g., AWS S3, GitHub, zenodo, Google Drive, personal host), some may require MD5, APIs in addition to URL. Also, one may take the maximum number of redirection as a parameter to avoid endless redirections.

Describe alternatives you've considered
If the node is spawn at where the data is located, such as deploying flame system on a phone while the training data is also on the same phone, then downloading may be unnecessary.

@GaoxiangLuo GaoxiangLuo added the enhancement New feature or request label Jul 6, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant