Select or Rewrite: Oracle-Guided Fine-Tuning for Generative ASR Post-Editing from N-Best Hypotheses
Large language models can post-edit ASR by using linguistic context to compare N-best hypotheses and rewrite the transcript when needed, often improving WER without retraining the recognizer.
ProofPaper ↗
Key points
- However, prompt-only correction is brittle: it may over-edit when top-1 is correct and hallucinate when the reference is absent, and often needs very large models.
- We propose oracle-guided fine-tuning that turns N-best lists into supervision via three curated pools: corrections (reference in the list but not top-1) to teach selection, regularizers (top-1 correct) to discourage unnecessary rewrites, and hard negatives (reference absent) to suppress hallucination.
- Fine-tuning on this data provides more effective supervision than random subset fine-tuning for learning a hybrid select-or-rewrite policy, with the clearest gains from around 3B parameters on the proprietary dataset but less conclusive results on CommonVoice, a divergence suggesting that public read-speech text may overlap with LLM pretraining and mask genuine post-editing gains.
Sources (1)
- [1]Select or Rewrite: Oracle-Guided Fine-Tuning for Generative ASR Post-Editing from N-Best HypothesesarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 10, 04:48 PM
Large language models can post-edit ASR by using linguistic context to compare N-best hypotheses and rewrite the transcript when needed, often improving WER without retraining the recognizer.
However, prompt-only correction is brittle: it may over-edit when top-1 is correct and hallucinate when the reference is absent, and often needs very large models.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 10, 2026Toward System One Clinical Intelligence: Parameter-Efficient Fine-Tuning and Counterfactual Explainability of Decision Foundation Models
- Oct 10, 2026Training-Free BioNEL-RAG: Advancing Biomedical Entity Linking through Retrieval-Augmented Generation and Large Language Model
- Oct 10, 2026Vela: User Behavior as Supervision Sparks LLM Intelligence
- Oct 9, 2026Embedding Graph Kernel Entropy (EGKE): Fine-grained Hallucination Detection for LLMs from Embeddings
- Oct 8, 2026Unsupervised Multimodal Reward Modeling
- Oct 8, 2026Understanding Jev, the new model everyone is talking about
