How Should a Camera-First AI Agent Work?

Key points

• Camera-first agents must act on the visual world as it unfolds.1
• We formalize this as Ephemeral-Target Interaction (ETI) and a Perceptual Deadline Contract separating a hard memory deadline from a soft reasoning deadline, grounded in the bounded, passive nature of sensory and working memory.1
• We propose a two-deadline architecture: an always-on, bounded, non-semantic fast path; an active remembering mechanism that commits a moment to durable memory before it is overwritten (event-driven in v1); and a deferred deliberation layer (multimodal model + tools).1
• We instantiate the contract on real MOT17 track windows, a consistency check, and calibrate the single-path baseline from measured frontier-model latency: even a short answer takes 1.2-9.3 s, and an agentic identify-and-search longer.1

Sources (1)

• [1]How Should a Camera-First AI Agent Work?
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 10, 09:53 AM

  We argue that the central obstacle for future camera-first agents is not recognition ability - multimodal models (vision-language and live/streaming alike) generalize well through language and remain the right tool for semantic reasoning - but time.

  Camera-first agents must act on the visual world as it unfolds.

Extractive summary: sentences quoted from the sources.