-
Notifications
You must be signed in to change notification settings - Fork 9
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Evaluation metrics #4
Comments
Good points. The default captions used
Which do you think is a better option? Also thanks for the pointer to ImageReward, I will look into it! |
I would also go for option 2 for now at least, because of what you said + the need to distribute the computation of all the metrics over the GPUs as well otherwise only rank zero would be used, while others GPUs would wait. |
Sounds good to me, it will take me some time to implement this. Let me know if you'd like to take some part of the PR. I see 3 direct parts:
I have partial implementations on all of these (except ImageReward) which I will push to a working branch soon that we could use as a starting point. |
I can take care of ImageReward, and help with others, so please go ahead and push the working branch so that I extend it. maybe you can do FID and I do CLIPScore, or the other way. |
Another work to consider: https://arxiv.org/abs/2305.01569, similar to ImageReward (they also compare themselves with ImageReward). Code: https://github.com/yuvalkirstain/PickScore |
Hi Mehdi, I have added some starting code in the |
Would be great to have (optional) model evaluation.
Possibilities:
The text was updated successfully, but these errors were encountered: