From Abstraction to Action: DSLs for Natural Language Abstraction Interpretation and Execution

Key points

• Natural-language instructions often rely on high-level abstractions, such as asking an agent to "stack blocks in a pyramid.1
• We demonstrate the efficacy of DSLs for NL interpretation on Hexagons, a visually-grounded instruction-following task graded for abstraction.1
• We further ask whether language models can induce such DSLs themselves from domain data.1
• Our results show that while code generation consistently improves over direct inference, code generation over a human-curated DSL yields dramatic gains, up to 10 absolute F1 points, reaching the human skyline, with gains concentrated at {\em higher} abstraction levels.1

Sources (1)

• [1]From Abstraction to Action: DSLs for Natural Language Abstraction Interpretation and Execution
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 02:51 PM

  To bridge the gap between natural language abstraction and concrete actions, we propose to borrow from coding environments the notion of a {\em domain-specific language} (DSL), an explicit representation layer whose documented primitives name recurring abstractions that expand into concrete actions.

  Natural-language instructions often rely on high-level abstractions, such as asking an agent to "stack blocks in a pyramid.

Extractive summary: sentences quoted from the sources.