LLM-based User-Product Relevance Evaluation for Dynamic Product Advertising

Key points

• Dynamic Product Advertising (DPA) retrieval systems are a core component of modern e-commerce advertising; however, their evaluation remains challenging.1
• The metric first compresses the engagement history into a compact set of intent representatives via Semantic-ID-grouped, event-stratified segmentation, then assesses the relevance of a recommended product to each individual intent and aggregates them as a user-level relevance score, yielding a stable and interpretable measure of retrieval quality.1
• To compute this metric at scale, we leverage large language models (LLMs) as semantic relevance judges, capturing nuanced functional and semantic relationships that heuristic methods often miss.1
• We propose an agentic, human-aligned prompting framework that generates multi-category relevance guidelines and distills teacher-model judgments into an efficient student model for scalable evaluation.1

Sources (1)

• [1]LLM-based User-Product Relevance Evaluation for Dynamic Product Advertising
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 10, 04:15 AM

  In this work, we propose a user--product relevance metric for offline evaluation of DPA retrieval systems.

  Dynamic Product Advertising (DPA) retrieval systems are a core component of modern e-commerce advertising; however, their evaluation remains challenging.

Extractive summary: sentences quoted from the sources.