Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrate DataStore with Payload entry #847

Open
mylibrar opened this issue Jul 1, 2022 · 1 comment
Open

Integrate DataStore with Payload entry #847

mylibrar opened this issue Jul 1, 2022 · 1 comment

Comments

@mylibrar
Copy link
Collaborator

mylibrar commented Jul 1, 2022

Is your feature request related to a problem? Please describe.
This is a follow-up issue to #840. After we restructure the payload system in #828, we have a new ontology called Payload and it's not entirely integrated to DataStore yet. Currently DataPack stores three payload lists for different modalities. We want to migrate them into DataStore without affecting the current interfaces.

Depends on

Describe the solution you'd like

  • We should maintain all payload lists in DataStore
    • __getitem__ should be efficient
  • Design how Payload's attributes can be stored in DataStore as primitive and built-in types.
    • Need to determine how to handle the dataclass and non-dataclass fields.
    • Remove __getstate__ and __setstate__ from Payload
  • We should keep the previous interfaces of accessing payloads, e.g., pack.text_payloads[text_payload_index]

Additional context

  • This is part of the data efficiency project
  • This PR should be made to the master branch.
@mylibrar
Copy link
Collaborator Author

#882 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant