Tootfinder

Evaluation Guidelines for Empirical Studies in Software Engineering involving LLMs
Sebastian Baltes, Florian Angermeir, Chetan Arora, Marvin Mu\~noz Bar\'on, Chunyang Chen, Lukas B\"ohme, Fabio Calefato, Neil Ernst, Davide Falessi, Brian Fitzgerald, Davide Fucci, Marcos Kalinowski, Stefano Lambiase, Daniel Russo, Mircea Lungu, Lutz Prechelt, Paul Ralph, Christoph Treude, Stefan Wagner

Evaluation Guidelines for Empirical Studies in Software Engineering involving LLMs
Large language models (LLMs) are increasingly being integrated into software engineering (SE) research and practice, yet their non-determinism, opaque training data, and evolving architectures complicate the reproduction and replication of empirical studies. We present a community effort to scope this space, introducing a taxonomy of LLM-based study types together with eight guidelines for designing and reporting empirical studies involving LLMs. The guidelines present essential (must) criteria…