perturber: A Stateless Service for Semantics-Preserving Text Perturbations

Key points

• Large language model (LLM) evaluations on clean benchmarks overstate model robustness: real inputs arrive perturbed, and these meaning-preserving variants routinely move model answers.1
• Studying this systematically requires applying perturbations one at a time, reproducibly, and identically wherever they run, yet existing tools couple them to a benchmarking stack rather than packaging them as a reusable transform layer.1
• It turns on a single primitive, with composition left to the caller chaining calls, so the service and the library produce identical output by construction.1
• We evaluate the tool itself, its correctness, reproducibility, and cost, rather than any model's robustness, measuring meaning-preservation per perturbation rather than assuming it.1

Sources (1)

• [1]perturber: A Stateless Service for Semantics-Preserving Text Perturbations
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 07:44 PM

  We present perturber, a stateless service and Python library that applies exactly one semantics-preserving perturbation per call, drawn from a first-party catalog of 68 perturbations across 11 families.

  Large language model (LLM) evaluations on clean benchmarks overstate model robustness: real inputs arrive perturbed, and these meaning-preserving variants routinely move model answers.

Extractive summary: sentences quoted from the sources.