Conditional information Retrieval for Longitudinal Survey Harmonisation

Key points

• Identifying conceptually equivalent questions across different surveys is a challenging information retrieval task where surface-level textual similarity is insufficient.1
• Two questions may be lexically similar yet assess entirely different sociological sub-concepts, or lexically different yet conceptually equivalent.1
• To reflect the realistic data scarcity conditions of specialised social science annotation, we use CLOSER as a case study.1
• Finally, we demonstrate that a model fine-tuned on a general social science thesaurus (ELSST) transfers robustly to task-specific survey data, offering a practical solution under data scarcity.1

Sources (1)

• [1]Conditional information Retrieval for Longitudinal Survey Harmonisation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 11:03 AM

  We introduce a conditional information retrieval framework that addresses this by decomposing the task into two steps: concept extraction, which identifies the underlying sociological concept of a survey question using a fine-tuned generative model, and concept-aware retrieval, which augments question representations with the extracted concept as an explicit conditioning signal for finer-grained a

  Identifying conceptually equivalent questions across different surveys is a challenging information retrieval task where surface-level textual similarity is insufficient.

Extractive summary: sentences quoted from the sources.