Tootfinder

When you're thinking that the phone didn't load the whole photo, then realize it's just snow at the lower half.

FATHOMS-RAG: A Framework for the Assessment of Thinking and Observation in Multimodal Systems that use Retrieval Augmented Generation
Samuel Hildebrand (Louisiana State University), Curtis Taylor (Oak Ridge National Lab), Sean Oesch (Oak Ridge National Lab), James M Ghawaly Jr (Louisiana State University), Amir Sadovnik (Oak Ridge National Lab), Ryan Shivers (Oak Ridge National Lab), Brandon Schreiber (Oak Ridge National Lab), Kevin Kurian (University of Florida)

FATHOMS-RAG: A Framework for the Assessment of Thinking and Observation in Multimodal Systems that use Retrieval Augmented Generation
Retrieval-augmented generation (RAG) has emerged as a promising paradigm for improving factual accuracy in large language models (LLMs). We introduce a benchmark designed to evaluate RAG pipelines as a whole, evaluating a pipeline's ability to ingest, retrieve, and reason about several modalities of information, differentiating it from existing benchmarks that focus on particular aspects such as retrieval. We present (1) a small, human-created dataset of 93 questions designed to evaluate a pipe…