New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Exploration Automation: Obtain Dataset, Select Tokenization, and Cache results #219

Open

gkielian opened this issue Aug 8, 2024 · 0 comments

Collaborator

gkielian commented Aug 8, 2024

Thinking we may want to have an option to set the dataset download directory to a standard directory in the home folder.

This way, one only needs to download the dataset, and perform tokenization once and save results.

This is for augmenting the run_experiments (and run_vizier.py) scripts to:

initialize download of the dataset if not already in expected location
tokenize the dataset if not already done (and also cache results for potential reuse)
adjust the run_experiments/run_vizier to take in special "tokenization" arguments for handling the prepare.py script.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment