Consider, Then Choose: Order-Invariant Decisions in System One Models

Key points

• For choice questions, this promise fails: the answer depends on the order of the options, increasingly so as they grow in number, and averaging over their cyclic shifts reduces this dependence without removing it.1
• Scoring each option alone, which the order cannot affect, shows that the order changes the answer mostly when the model is unsure among its top few options, even though the listwise answer looks confident.1
• We therefore propose $Runoff$, which considers, then chooses: it scores each option alone to keep a few candidates, then compares them in listings where each candidate takes every position.1
• On two open models and five datasets, it changes no answer under shuffling, against 9.3% for cyclic averaging, and with 20 options it is about as accurate and three to four times faster.1

Sources (1)

• [1]Consider, Then Choose: Order-Invariant Decisions in System One Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 12:02 PM

  System One models answer typed questions in a single forward pass and promise the same answer for the same input.

  For choice questions, this promise fails: the answer depends on the order of the options, increasingly so as they grow in number, and averaging over their cyclic shifts reduces this dependence without removing it.

Extractive summary: sentences quoted from the sources.