LG-Bench: A Graph-Structured Evaluation Benchmark for Life Sciences

Key points

• Traditional evaluation benchmarks reduce inherently interconnected scientific knowledge in life sciences into flat lists of questions, disregarding the underlying topological structure of the knowledge.1
• Our approach constructs a weighted evaluation graph using bidirectional matching and semantic similarity algorithms, where nodes represent questions and edge weights capture their semantic relationships.1
• The Global Coherence Score (GCS) measures a model's consistency within semantically related neighborhoods, while Knowledge Balance Score (KBS) analyzes how model errors are distributed across the graph to reveal conceptual blind spots.1
• LG-Bench facilitates fine-grained comparison of LLMs by surfacing differences in conceptual coherence and patterns of knowledge organization across models.1

Sources (1)

• [1]LG-Bench: A Graph-Structured Evaluation Benchmark for Life Sciences
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 05:38 PM

  We introduce, the first graph-structured benchmark for life sciences, featuring over 10,000 high-quality multiple-choice questions across medicine, biology, and chemistry.

  Traditional evaluation benchmarks reduce inherently interconnected scientific knowledge in life sciences into flat lists of questions, disregarding the underlying topological structure of the knowledge.

Extractive summary: sentences quoted from the sources.