Web Scale on a Budget: Tail-Aware Retrieval Evaluation

Key points

• A common practice in instance retrieval is to grow the gallery size to improve a benchmark's discrimination capability.1
• We define the per-query retrieval hardness, a gallery-scale agnostic quantity that yields a retrieval metric at any scale, and demonstrate it can be derived from a small random sample.1
• The underlying sampled-rank machinery is long established in recommender-system and biometric evaluation, but to our knowledge has never been adapted to visual retrieval.1
• We release code and the per-query hardness to facilitate faster, cheaper, and greener retrieval evaluation.1

Sources (1)

• [1]Web Scale on a Budget: Tail-Aware Retrieval Evaluation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 08:54 AM

  We show the web-scale indexing price is largely avoidable: evaluation itself obeys a scaling law.

  A common practice in instance retrieval is to grow the gallery size to improve a benchmark's discrimination capability.

Extractive summary: sentences quoted from the sources.