Typed Decision Models under Prevalence Shift: Decision Training, Calibration, and Implicit Priors

Key points

• We ask whether decision training improves a model over its own backbone, and whether a TDM's calibration and thresholds withstand prevalence (label) shift as well as a fine-tuned classifier's.1
• We compare three TDMs (Jev, Tev1-4B, Nimble-9B), their base models and two trained classifiers on identical decisions from five public datasets under simulated shift, with 35 paired tests fixed before any test was computed.1
• First, decision training is not uniformly beneficial: Nimble-9B, which differs from Qwen3.5-9B only by a LoRA adapter, is better calibrated on intent tasks but 12.0 Macro-F1 points worse on financial sentiment.1
• Second, under moderate shift Jev's expected calibration error (ECE) and threshold risk grow significantly more than a fine-tuned DeBERTa's only on financial sentiment, by 1.7 and 2.5 percentage points; averaged over 2,000 shift scenarios (exploratory), the gap is at most one point on every dataset.1

Sources (1)

• [1]Typed Decision Models under Prevalence Shift: Decision Training, Calibration, and Implicit Priors
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 01:01 PM

  Typed decision models (TDMs) return a probability over caller-defined options, and Jev is described by its vendor as calibrated.

  We ask whether decision training improves a model over its own backbone, and whether a TDM's calibration and thresholds withstand prevalence (label) shift as well as a fine-tuned classifier's.

Extractive summary: sentences quoted from the sources.