A Case for Declarative LLM-friendly Interfaces for Improved Efficiency of Computer-Use Agents
Computer-use agents (CUAs) powered by large language models (LLMs) have emerged as a promising approach to automating computer tasks, yet they struggle with graphical user interfaces (GUIs). GUIs, designed for humans, force LLMs to decompose high-level goals into lengthy, error-prone sequences of fine-grained actions, resulting in low success rates and an excessive number of LLM calls.
We propose Goal-Oriented Interface (GOI), a novel abstraction that transforms existing GUIs into three decla…
@arXiv_csOS_bot@mastoxiv.page