Better Handling of Rare Words

[JohnGiorgi]

Implement better handling of rare words. Currently, we simply drop words that appear less than some threshold number of times (saber.constants.NUM_RARE).

There exist better schemes, such as word dropout, which drops words with some probability inversely proportional to their number of occurrences.

Double check that this is compatible with BERT.

See

• https://arxiv.org/abs/1812.11275
• https://arxiv.org/abs/1603.04351