Normalization during retrieval scores computation

[violenil]

Hi! Loving the Arena for quick inspection of models :)

I noticed that the scores for the retrieval are computed as dot products, as opposed to cosine similarity, even though the embeddings are not normalized. I manually added normalization during a local deployment and got significantly different results, at least for the jinaai/jina-embeddings-v2-base-en model. Do you think we can add an optional parameter to the model_meta.yml to normalize embeddings during the model.encode call? I'm happy to make a PR.

[Muennighoff]

Thanks! For the arena that's live we're actually using the GCP index which normalizes first & then does dot product i.e. cosine similarity:

arena/retrieval/gcp_index.py

Line 160 in 64a8780

feature_norm_type="UNIT_L2_NORM",

We should definitely adapt it for the local index though, but I think it should be done in the models folder i.e. here: https://github.com/embeddings-benchmark/mteb/tree/main/mteb/models

I think we should probably add Jina in this file: https://github.com/embeddings-benchmark/mteb/blob/main/mteb/models/sentence_transformers_models.py & then activate normalization there so it always uses normalization when loaded via mteb.get_model(... - cc @KennethEnevoldsen

[KennethEnevoldsen]

Yep, that is totally correct - the https://github.com/embeddings-benchmark/mteb/blob/main/mteb/models/ folder is the gold standard reference for evaluated models.