Evaluating Semantic Relation Knowledge through Bidirectional Relation Prediction from Gloss Pairs
In order to evaluate semantic relation knowledge of large language models (LLMs), we propose a challenging task in which an LLM predicts the relation between two glosses (definitions of senses) presented in two different orders.
ProofPaper ↗
Key points
- Semantic relations organise word senses and are fundamental to lexical semantics.
- The predictions for the two orders are evaluated for consistency with the symmetry or asymmetry of semantic relations.
- Our task is designed for both monolingual and cross-lingual evaluation, allowing us to examine whether relation knowledge remains accessible when the same senses are verbalised through glosses in different languages.
- Our results suggest limitations in current LLMs' semantic relation knowledge, both in identifying relations and in respecting asymmetry.
Sources (1)
- [1]Evaluating Semantic Relation Knowledge through Bidirectional Relation Prediction from Gloss PairsarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 10, 01:16 AM
In order to evaluate semantic relation knowledge of large language models (LLMs), we propose a challenging task in which an LLM predicts the relation between two glosses (definitions of senses) presented in two different orders.
Semantic relations organise word senses and are fundamental to lexical semantics.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 10, 2026UltraBERT: Beyond the Limit of BERT Pre-Training
- Oct 10, 2026ProgAgent: Learning Tool Orchestration through Programmatic Training
- Oct 10, 2026Vela: User Behavior as Supervision Sparks LLM Intelligence
- Oct 9, 2026Embedding Graph Kernel Entropy (EGKE): Fine-grained Hallucination Detection for LLMs from Embeddings
- Oct 9, 2026Surfing the Activation Wave: Analyzing the Information and Geometry of LLM Hidden States for Speculative Decoding
- Oct 9, 2026ClawLoop: Self-Improving Agents via Harness-Native Training Loop