Tootfinder

Deprecating Benchmarks: Criteria and Framework
Ayrton San Joaquin, Rokas Gipi\v{s}kis, Leon Staufer, Ariel Gil
https://arxiv.org/abs/2507.06434 https://

Deprecating Benchmarks: Criteria and Framework
As frontier artificial intelligence (AI) models rapidly advance, benchmarks are integral to comparing different models and measuring their progress in different task-specific domains. However, there is a lack of guidance on when and how benchmarks should be deprecated once they cease to effectively perform their purpose. This risks benchmark scores over-valuing model capabilities, or worse, obscuring capabilities and safety-washing. Based on a review of benchmarking practices, we propose criter…

Efficiently Ranking Software Variants with Minimal Benchmarks
Th\'eo Matricon, Mathieu Acher, Helge Spieker, Arnaud Gotlieb
https://arxiv.org/abs/2509.06716 https://

Efficiently Ranking Software Variants with Minimal Benchmarks
Benchmarking is a common practice in software engineering to assess the qualities and performance of software variants, coming from multiple competing systems or from configurations of the same system. Benchmarks are used notably to compare and understand variant performance, fine-tune software, detect regressions, or design new software systems. The execution of benchmarks to get a complete picture of software variants is highly costly in terms of computational resources and time. In this pape…

Artificial Analysis' benchmarks show Grok 4 is the leading AI model, a first for xAI, and its per-token pricing is more expensive than Gemini 2.5 Pro and o3 (@artificialanlys)
https://x.com/artificialanlys/status/1943166841150644622

Artificial Analysis (@ArtificialAnlys) on X
xAI gave us early access to Grok 4 - and the results are in. Grok 4 is now the leading AI model. We have run our full suite of benchmarks and Grok 4 achieves an Artificial Analysis Intelligence Index of 73, ahead of OpenAI o3 at 70, Google Gemini 2.5 Pro at 70, Anthropic Claude

SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge
Lukas Haas, Gal Yona, Giovanni D'Antonio, Sasha Goldshtein, Dipanjan Das
https://arxiv.org/abs/2509.07968

SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge
We introduce SimpleQA Verified, a 1,000-prompt benchmark for evaluating Large Language Model (LLM) short-form factuality based on OpenAI's SimpleQA. It addresses critical limitations in OpenAI's benchmark, including noisy and incorrect labels, topical biases, and question redundancy. SimpleQA Verified was created through a rigorous multi-stage filtering process involving de-duplication, topic balancing, and source reconciliation to produce a more reliable and challenging evaluation set, alongsi…

I benchmarked #PHP's native serializer vs code export. You won't believe what I found!
https://peakd.com/hive-168588/@crell/benchmarking-serialization

Benchmarking serialization | PeakD
I benchmarked PHP's native serializer vs code export. You won't believe what I found!... by crell

Benchmarking Single-Qubit Gates on a Neutral Atom Quantum Processor
Artem Rozanov, Boris Bantysh, Ivan Bobrov, Gleb Struchalin, Stanislav Straupe
https://arxiv.org/abs/2509.06881

Benchmarking Single-Qubit Gates on a Neutral Atom Quantum Processor
We present benchmarking results for single-qubit gates implemented on a neutral atom quantum processor using Direct Randomized Benchmarking (DRB) and Gate Set Tomography (GST). The DRB protocol involves preparing stabilizer states, applying $m$ layers of native single-qubit gates, and measuring in the computational basis, providing an efficient error characterization under a stochastic Pauli noise model. GST enables the full, self-consistent reconstruction of quantum processes, including gates,…

A new day a new AI benchmark.
#ai #benchmarks

OpenAI's o3 tops new AI league table for answering scientific questions
SciArena uses votes by researchers to evaluate large language models’ responses on technical topics.

It’s exactly four weeks ago today that the Jeffrey Epstein story broke,
or re-broke in its current form.
On Friday, July 11, the world learned of the tense meeting that took place at the White House that previous Wednesday,
in which FBI Deputy Director Dan Bongino clashed with Attorney General Pam Bondi over the handling of the Epstein files.
Bongino was so incensed that he didn’t go to work that Friday
and threatened to resign.
He has, at least for now…

On Epstein, Even MAGA World Can Smell That Trump Is Hiding Something
We’re four weeks into the current iteration of the Epstein scandal. And there is no way it’s going away.

DischargeSim: A Simulation Benchmark for Educational Doctor-Patient Communication at Discharge
Zonghai Yao, Michael Sun, Won Seok Jang, Sunjae Kwon, Soie Kwon, Hong Yu
https://arxiv.org/abs/2509.07188 …

DischargeSim: A Simulation Benchmark for Educational Doctor-Patient Communication at Discharge
Discharge communication is a critical yet underexplored component of patient care, where the goal shifts from diagnosis to education. While recent large language model (LLM) benchmarks emphasize in-visit diagnostic reasoning, they fail to evaluate models' ability to support patients after the visit. We introduce DischargeSim, a novel benchmark that evaluates LLMs on their ability to act as personalized discharge educators. DischargeSim simulates post-visit, multi-turn conversations between LLM-…

CoreCodeBench: A Configurable Multi-Scenario Repository-Level Benchmark
Lingyue Fu, Hao Guan, Bolun Zhang, Haowei Yuan, Yaoming Zhu, Jun Xu, Zongyu Wang, Lin Qiu, Xunliang Cai, Xuezhi Cao, Weiwen Liu, Weinan Zhang, Yong Yu
https://arxiv.org/abs/2507.05281

CoreCodeBench: A Configurable Multi-Scenario Repository-Level Benchmark
As Large Language Models (LLMs) demonstrate increasingly sophisticated code processing capabilities, evaluating their performance on engineering-level code remains challenging. Existing repository-level benchmarks primarily focus on single scenarios, such as code generation or bug fixing, without adequately capturing the diversity and complexity of real-world software or project engineering workflows. Furthermore, these benchmarks suffer from limited controllability in question positioning and …