When PTQ Calibration Helps or Hurts: Dose-Driven Collapse and Validation-Gated Abstention in Quantized Text Embedders

Key points

• We ask two questions that are usually conflated: how far to calibrate, and whether to calibrate at all.1
• Across three embedders (e5-base-v2, BGE-M3, EmbeddingGemma-300m) at 4-bit weight quantization, 161 calibration runs, the outcome of a fixed procedure ranges from a gain to catastrophic degradation, and the driver is the dose of the update, measured as the functional displacement Dfunc of the quantized model from its uncalibrated start.1
• On EmbeddingGemma the same learning-rate grid gives +1.47 to -71.53 points of in-domain NDCG@10 change; a label-free cosine objective collapses at the same dose (-69.52 vs -71.63), so the collapse is a property of learnable calibration rather than of retrieval supervision.1
• Objective sensitivity is heterogeneous: on e5, ranking calibration is substantially more robust than cosine at matched displacement (+14.16 +/- 4.34 pp, 95% t-CI [+3.37, +24.96], three paired slates), whereas on BGE-M3 the ordering is directionally opposite but unresolved ([-5.18, +2.38]).1

Sources (1)

• [1]When PTQ Calibration Helps or Hurts: Dose-Driven Collapse and Validation-Gated Abstention in Quantized Text Embedders
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 11:43 PM

  Learned post-training quantization (PTQ) calibration fits quantizer parameters on a small calibration set, and for a text embedder the natural supervision is retrieval relevance.

  We ask two questions that are usually conflated: how far to calibrate, and whether to calibrate at all.

Extractive summary: sentences quoted from the sources.