Random Retrieval Scores

[ra1ph2]

Hey, I have downloaded the YouCook2 features and am training a common space learning network to associate positive sentence-clip pairs for video retrieval.
I have loaded the video features and the text features from the pickle files using a custom dataset and then am loading the files using the dictionary provided.
I have checked my code for errors and am pretty sure that it is correct since when I use the same training or val set as the training set and then check retrieval scores on it, the model is converging to a good score. But when I validate the model on an unseen set the results are totally random (Eg- MedianR being 1500/3000).
So, I just wanted to know if there is some trivial step that I might missing while loading the dataset or if the sentence-pairs are incorrectly matched in the pickle file.
Thanks in advance and I hope I am clear in explaining the issue.

[RSteendam]

I'm also curious to this. @ra1ph2 Did you find out what could be causing this?