Can LLM Agents Efficiently Identify Appliance Faults? Insights From Diagnosing a Simulated Appliance

Key points

• In so-called "Industry 4.0," integrated sensors, analytics tools, and other digital technologies supplement industrial workflows, potentiating enhanced visibility and automation.1
• Prognostics and health management (PHM) and natural language processing (NLP) practitioners alike have explored the application of large language models (LLMs) to industrial asset management, but many prior works' focus on exam-like benchmarks has limited their assessment of LLMs' ability to automate practical operations.1
• Our subsequent analyses, therefore, focus on: differentiating models by diagnostic efficiency (RQ1); examining the lagging performance of the three worst models (RQ2); and identifying viable heuristics for selective prediction (RQ3), including that initial diagnostic uncertainty and LLM-judged soundness of reasoning are high-signal predictors of diagnostic success.1
• In addition to our benchmark results (Section 3.1), we provide practical insights for improving the performance of the open-weight models on CoffeeBench, such as by explicitly tracking episode state on their behalf to reduce timeouts (Section 3.2) or applying a deliberation loop to refine diagnostic hypotheses (Section 3.3).1

Sources (1)

• [1]Can LLM Agents Efficiently Identify Appliance Faults? Insights From Diagnosing a Simulated Appliance
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 11:12 PM

  Herein, we introduce CoffeeBench, a fault diagnostics task for LLM agents, using a simulated appliance.

  In so-called "Industry 4.0," integrated sensors, analytics tools, and other digital technologies supplement industrial workflows, potentiating enhanced visibility and automation.

Extractive summary: sentences quoted from the sources.