ResearchResearch paperTraining & Scaling · Large Language Models1 source · Oct 16, 2026

ToPA: Block-wise Toeplitz Adaptation for Expressive and Efficient Fine-Tuning

Parameter-Efficient Fine-Tuning (PEFT) has emerged as a prevalent strategy for adapting large pre-trained models.

To overcome these limitations, we explore the use of Toeplitz matrices, whose entries are constant along each diagonal, therefore providing a compact parameterization without imposing explicit rank or sparsity constraints.
To address this issue, we propose ToPA, a more refined version of this idea that replaces standard Toeplitz matrices with block-wise Toeplitz, greatly increasing each factor's capacity and shortening the chain, thereby stabilizing training and improving expressiveness.
We theoretically show that products of block-wise Toeplitz matrices can approximate arbitrary matrices under shorter chain length, justifying the structural design of ToPA.
Furthermore, we demonstrate that ToPA offers greater expressivity than existing low-rank and sparse parameterizations.

[1]ToPA: Block-wise Toeplitz Adaptation for Expressive and Efficient Fine-Tuning
arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 03:38 AM
Parameter-Efficient Fine-Tuning (PEFT) has emerged as a prevalent strategy for adapting large pre-trained models.
To overcome these limitations, we explore the use of Toeplitz matrices, whose entries are constant along each diagonal, therefore providing a compact parameterization without imposing explicit rank or sparsity constraints.

Extractive summary: sentences quoted from the sources.

Before this