North or South? Diagnosing LLMs' Understanding of North Korean Language, Society, and Knowledge

Key points

• Large Language Models (LLMs) have made substantial progress in multilingual question answering, yet their ability to understand low-resource sociolinguistic contexts remains under-explored.1
• We study North Korea as a unique testbed where decades of division have produced distinct differences from South Korea in vocabulary, orthography, norms, and ideological discourse.1
• Evaluating diverse recent LLMs, we find a modest performance drop on North Korean language inputs, a larger gap in sociocultural perspective alignment, and severe limitations on domain-specific knowledge.1
• Our results demonstrate that linguistic similarity alone does not guarantee sociocultural or domain-specific understanding, highlighting the critical need for grounded evaluation resources for underrepresented cultural and political contexts.1

Sources (1)

• [1]North or South? Diagnosing LLMs' Understanding of North Korean Language, Society, and Knowledge
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 11, 02:26 AM

  We introduce LENS (Language and sociocultural Evaluation for North Korean Society), a human-validated benchmark of 1,765 questions evaluating LLMs across three dimensions: linguistic robustness, sociocultural perspective alignment, and domain-specific knowledge.

  Large Language Models (LLMs) have made substantial progress in multilingual question answering, yet their ability to understand low-resource sociolinguistic contexts remains under-explored.

Extractive summary: sentences quoted from the sources.