KiranaBench: Can a Language Model Run a Neighbourhood Shop?

Key points

• Existing benchmarks for this setting are almost exclusively English-language, Western in retail context, and single-agent, with the agent observing the business directly.1
• We introduce KiranaBench, a simulated Indian general store in which four roles (manager, cashier, helper, and bookkeeper) are each played by a separate LLM agent.1
• Evaluating four open-weight LLMs, we find that all fall far short of a fixed-rule shopkeeper baseline: the best model earns only one-eighth of the baseline's profit, and two models finish below an unmanaged shop.1
• KiranaBench offers evidence that current LLM agents are not yet reliable for autonomously operating small businesses in India, and provides a testbed for developing agents suited to settings that are long-horizon, collaborative, and multilingual.1

Sources (1)

• [1]KiranaBench: Can a Language Model Run a Neighbourhood Shop?
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 06:02 AM

  Large language model (LLM) agents are increasingly evaluated on long-horizon tasks, including autonomously running a business.

  Existing benchmarks for this setting are almost exclusively English-language, Western in retail context, and single-agent, with the agent observing the business directly.

Extractive summary: sentences quoted from the sources.