Evaluating Large Language Models for Multilingual Abstractive Summarization: A Comprehensive Study of English and Bengali

Key points

• We present a standardized zero-shot benchmark of eight closed-source and open-weight LLMs on size-matched English and Bengali XL-Sum test sets.1
• Our evaluation combines lexical and semantic metrics with reference-free factuality verification and multidimensional LLM-as-a-Judge assessment, providing a comprehensive analysis of both surface-level quality and factual reliability in multilingual summarization.1
• Results reveal substantial cross-lingual variation: GPT-5.6 Luna leads several English lexical metrics but does not maintain the same advantage in Bengali, where LLaMA 3.3 70B achieves the highest ROUGE and BLEU scores.1
• These findings demonstrate that strong English summarization performance does not necessarily transfer to Bengali and emphasize the need for comprehensive evaluation frameworks for low-resource languages.1

Sources (1)

• [1]Evaluating Large Language Models for Multilingual Abstractive Summarization: A Comprehensive Study of English and Bengali
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 06:05 PM

  Abstractive summarization remains challenging across languages, where linguistic quality and factual consistency may diverge, particularly in low-resource settings.

  We present a standardized zero-shot benchmark of eight closed-source and open-weight LLMs on size-matched English and Bengali XL-Sum test sets.

Extractive summary: sentences quoted from the sources.