How to combine resumable IterableDataset with accelerate & transformers

[yzhangcs]

Hi @lhoestq , thank you for developing the great feature of state_dict for IterableDataset, which I found to be vital for resuming training with huge datasets from checkpoints.
However, I've noticed that it's currently not well-supported by the accelerate and transformers libraries, as they require native PyTorch dataloaders and datasets.
This makes it challenging to use IterableDataset and StatefulDataLoader (as demonstrated in the tutorials) in combination with accelerate and transformers.
I'm wondering if there are any plans or possibilities to improve the compatibility of those with these libraries.

[lhoestq]

yep, stay tuned 👀 😄