Uniform Information Density-Based Preference Optimization

Key points

• In this study, we adopt a well-established psycholinguistic theory, Uniform Information Density, to develop a novel preference optimization algorithm, UIDPO.1
• The basic assumption is that humans prefer to produce language in which information is uniformly distributed, because uniformity induces less cognitive processing effort.1
• To implement this idea into the algorithm, we derive a compound metric for information uniformity by combining the global standard deviation and the local second-order difference, and integrate it to the objective of preference optimization.1
• Additional linguistic analysis on complement clauses also shows that UIDPO can better reflect the tendencies of keeping information uniform in natural language than baselines.1

Sources (1)

• [1]Uniform Information Density-Based Preference Optimization
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 10, 04:55 AM

  To enable large language models to generate outputs that align with human preferences, optimization algorithms based on the comparison between chosen and rejected output pairs have been developed and been proven effective.

  In this study, we adopt a well-established psycholinguistic theory, Uniform Information Density, to develop a novel preference optimization algorithm, UIDPO.

Extractive summary: sentences quoted from the sources.