It's been pointed out by several people that Adam usually works much better if the learning rate is started low and raised to a final value with an e.g. raised cosine learning rate schedule, to allow the model to learn a rough distribution of moments before starting the updates.
The lack of this startup schedule is the cause of the issue where initial updates look "binary", which can sometimes be problematic enough to derail an entire reconstruction. It should be a pretty simple addition
It's been pointed out by several people that Adam usually works much better if the learning rate is started low and raised to a final value with an e.g. raised cosine learning rate schedule, to allow the model to learn a rough distribution of moments before starting the updates.
The lack of this startup schedule is the cause of the issue where initial updates look "binary", which can sometimes be problematic enough to derail an entire reconstruction. It should be a pretty simple addition