Pretraining Code for Beats

[EssamWisam]

As was illustrated in the paper, the training mechanism for beats involves distillation, iterates between two models and changes the tokenizer architecture after the first iteration which makes nontrivial to implement pretraining.

Could you please share the pretraining code? We are interested in pretraining on a dataset that is majorly different from audio set.

Thanks.