Report Adjusted Mean Rank (AMR)

[cthoyt]

The the adjusted mean rank (AMR) complements the reported MRR and hits@k metrics - it has been demonstrated to provide more fair comparisons between datasets and varying testing set sizes. It would be great if LibKGE could report this metric in its ensemble.

Citation:

Max Berrendorf, Evgeniy Faerman, Laurent Vermue, and Volker Tresp (2020) Interpretable and Fair Comparison of Link Prediction or Entity Alignment Methods with Adjusted Mean Rank. arXiv, 2002.06914.

[samuelbroscheit]

I saw that paper a while ago and I really like it (esp. also the Appendix). I think more metrics are welcome.  Could you provide a PR for this feature?

[cthoyt]

I did a bit of digging in the code to see where this might fit. I'm not sure it will be so easy for me to make a contribution

[rgemulla]

The AMR is a rescaling of the mean rank by a test set-dependent constant (so, nothing will change qualitatively for a fixed test set). To report it, we'd need to compute the average number of non-filtered candidates during evaluation, and then divide the mean rank by half of that. This probably should be done by extending

kge/kge/job/entity_ranking.py

Line 477 in e7bd9b7

def _filter_and_rank(

to also report the number of filtered subjects and objects. From this, we can also compute the adjusted mean rank index.