How to compute MRR @n?

Question

Supposed given a query, our system return 10 recommendation result

doc1, doc2, ... doc10

if the ground truth says relevant document says the relevant document is doc100 (which is in position 100), does the reciprocal rank should be 1/100 ? I think it's hard to push the system to return 100 document.

But if i only include recommendation result which had the relevant document in position below top10, I think that is a Naive approach. And i don't really understand the concept of MRR@10, MRR@5, etc.

Can someone please explain to me? Thank you

score 2 · Accepted Answer · answered Aug 05 '22 at 21:26

2

The MRR in that case is defined to be zero.

The reasoning is this: imagine you didn’t have any more documents; how would you tell the difference between it being on the next page or not being returned at all?

The goal of these low rank MRR is to model the first page or a website that can only fit three recommendations. To the user there’s no difference between documents that appear just after the cutoff and a hundred ranks deeper.

answered Aug 05 '22 at 21:26

John Foley

957
9
19

thank you for your explanation. But is that okay if when the result is not in top 3 and i assume the MRR is 1/4? (I assume the relevant document is on position 4) – Muhammad Fhadli Aug 08 '22 at 08:35
No, in that case you have to use zero. – John Foley Aug 09 '22 at 10:45

How to compute MRR @n?

1 Answers1