LLMs program using Reusable and Composable Abstract Representations

Key points

• When writing code, the same computation can be expressed in different programming languages and through different implementation strategies.1
• Using activation steering, we show that function, programming language, and implementation strategy can be independently manipulated.1
• We additionally find that finetuning on examples of arbitrarily defined functions in one programming language transfers to other languages, consistent with the models reusing abstract features of code generation that are shared across different examples.1
• Finally, to study how such abstract representations come to be, we implement a minimal model of shared, reusable, and composable structure in a synthetic setting, and show that transformers trained on such data learn representations that reuse these parts flexibly across contexts.1

Sources (1)

• [1]LLMs program using Reusable and Composable Abstract Representations
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 09:56 PM

  We show that the activations of LLMs capture this shared structure in reusable and composable abstract representations when they generate code.

  When writing code, the same computation can be expressed in different programming languages and through different implementation strategies.

Extractive summary: sentences quoted from the sources.