What Do RTL Repair Benchmarks Actually Measure?

Key points

• To this end, we construct and hash-bind RTLRepairDataset, 5,784 (broken, fixed) training pairs; RTLRepairBench, 459 held-out cases with 374 syntactic and 85 behavioral mutations; and an evaluation harness supporting simulation and formal equivalence.1
• First, simulation and formal scoring give opposite signs for the apparent effect of providing a specification in one evaluated cohort; reset-aware rescoring attributes almost all of the simulation-formal disagreement to reset and first-cycle comparison semantics.1
• Second, on 274 mined model failures, the specification-only minus specification-conditioned recovery point estimate ranges from -8.5 to +18.3 percentage points across six models.1
• We translate these findings into evaluation controls: report initialization, reset, first-cycle, and unknown-value semantics; disclose oracle coverage and undecided cases; and include a specification-only baseline.1

Sources (1)

• [1]What Do RTL Repair Benchmarks Actually Measure?
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 05:45 PM

  RTL repair benchmarks should measure whether a model can fix broken RTL given a specification.

  To this end, we construct and hash-bind RTLRepairDataset, 5,784 (broken, fixed) training pairs; RTLRepairBench, 459 held-out cases with 374 syntactic and 85 behavioral mutations; and an evaluation harness supporting simulation and formal equivalence.

Extractive summary: sentences quoted from the sources.