handling access authentication for datasets

[luisoala]

situation: many datasets are openly available but require some type of authentication (create account, etc..) before download is possible.

question: how can we let users of datasets know?

possible solution: allow creators of croissant datasets to add some "authentication_information" field where they write down in natural language how the authentication is done.

[marcenacp]

hey @luisoala, we support basic authentication (see: Loading a distribution via HTTP with Basic Auth).

on top of that, we could allow manual downloads in mlcroissant. Meaning you could input a hashmap filename->filepath on disk where you manually downloaded the data. that would allow to handle more complex use cases where an authentication is needed.

WDYT?