Lüshi-1840: Building and Probing a Knowledge Cutoff for a Classical Chinese Language Model

Key points

• Language models trained only on text from before a given year are useful for studying hindsight and historical language, but the open collections of pre-modern Chinese carry no publication dates and unreliable dynasty labels.1
• A first corpus, dated by a common heuristic, in effect ended at 1644 while containing twentieth-century material.1
• On the model side we measure surprisal, relative to a modern reference, on annals compiled after the cutoff.1
• For events after 1840 against events up to 1820 the ratio is 1.38 (95% interval 1.33-1.44), against about 1.00 for modern general-purpose models.1

Sources (1)

• [1]Lüshi-1840: Building and Probing a Knowledge Cutoff for a Classical Chinese Language Model
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 12, 05:31 AM

  We present Lüshi-1840, a 983M-parameter model trained from scratch on 12,262 books (1.28B characters) selected as composed before 1840.

  Language models trained only on text from before a given year are useful for studying hindsight and historical language, but the open collections of pre-modern Chinese carry no publication dates and unreliable dynasty labels.

Extractive summary: sentences quoted from the sources.