Evaluating Bias in Chinese Large Language Models across Tasks and Registers: Task Operationalization Effects and Irony Classification Risks

Key points

• We conducted a preregistered multi-task behavioral audit of multiple Chinese large language models.1
• In the content classification experiment, the core panel's classification error rate rose from 10.9% in the formal register to 23.5% in the yinyang (an echoic-irony register) version, a risk difference of +12.7 percentage points.1
• Explicit irony-aware prompts reduced the intervention panel's error rate by 9.6 percentage points.1
• Thus, when evaluating large language models for Chinese content governance, the effect of task operationalization on bias readings is an empirical question that requires data to answer, and safety evaluation should incorporate the ironic register as a test condition.1

Sources (1)

• [1]Evaluating Bias in Chinese Large Language Models across Tasks and Registers: Task Operationalization Effects and Irony Classification Risks
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 10, 01:41 AM

  In Chinese content governance, large language models are increasingly used to identify toxic and discriminatory expressions, yet both task operationalization and ironic expression can alter bias measurement readings.

  We conducted a preregistered multi-task behavioral audit of multiple Chinese large language models.

Extractive summary: sentences quoted from the sources.