-
|
i remembered someone worked on a pt adapter to load croissant into pt dataset object. searching the issues and branches and couldnt find it. any hints? |
Beta Was this translation helpful? Give feedback.
Replies: 3 comments
-
|
@luisoala There is currently a minimal notebook for an adapter using FLORES-200. I am planning to expand on the core functionality of the adapter in future patches. Some of the obvious pain points are 1) multiprocessing and 2) handling of complex object types (e.g., images, video), which have their own semantics and performance issues. Let me know if you have any thoughts :) |
Beta Was this translation helpful? Give feedback.
-
|
thx michael! ill take a look at it next week. that looks promising. let me know if i can support, im interested, also in the performance direction |
Beta Was this translation helpful? Give feedback.
-
|
FYI, the TFDS recipe also uses a DataLoader with pre-prepared data and showcases an end-to-end training with torch and torch.data. |
Beta Was this translation helpful? Give feedback.
@luisoala There is currently a minimal notebook for an adapter using FLORES-200. I am planning to expand on the core functionality of the adapter in future patches. Some of the obvious pain points are 1) multiprocessing and 2) handling of complex object types (e.g., images, video), which have their own semantics and performance issues. Let me know if you have any thoughts :)