Tootfinder

FormGym: Doing Paperwork with Agents
Matthew Toles, Rattandeep Singh, Isaac Song Zhou Yu
https://arxiv.org/abs/2506.14079 https://arx…

FormGym: Doing Paperwork with Agents
Completing paperwork is a challenging and time-consuming problem. Form filling is especially challenging in the pure-image domain without access to OCR, typeset PDF text, or a DOM. For computer agents, it requires multiple abilities, including multi-modal understanding, information retrieval, and tool-use. We present a novel form-filling benchmark consisting of 432 fields spread across 55 documents and 3 tasks, requiring knowledge of 236 features per user. We find that baseline VLAs achieve les…