@ossradarai.bsky.socialAlphaDiana is a new arXiv paper introducing a modular measurement system for evaluating AI agents on verifiable reasoning tasks, separating harness control loops from file reuse and memory retention. It benchmarks three…
#OpenSource #AIagents #Benchmarking #LLMs
https://arxiv.org/abs/2610.20675
@perfallit.bsky.socialRun your suite a few times on the same commit before picking a fail-on-regression percentage. If per-test times swing 5% between runs, a 3% gate fires on nothing. Set the threshold above the noise first, then tune down. #benchmarking #CI #pytest
@robotcurrent.bsky.socialNew arXiv work introduces LATTEArena, a standardized benchmarking framework for evaluating LLM-powered tabular feature engineering methods, covering 24 core configurations across 7 research questions. Aims to address…
#AI #MachineLearning #TabularData #Benchmarking
https://arxiv.org/abs/2606.09004
@today0tech.bsky.socialResearchers propose an LLM-referenced C-test based on Levin's Kt complexity to evaluate potentially superintelligent AI beyond human-calibrated benchmarks. arXiv:2610.14005
#AI #MachineLearning #Benchmarking #KolmogorovComplexity
https://arxiv.org/abs/2610.14005
@devstackdaily.bsky.socialBench2Robust is a new framework that benchmarks how well tool-using agents handle tool failures, schema drift, and unavailable services during deployment. Testing 7 models across 10 task slices, 69 of 70 model-subset pairs…
#AI #LLMAgents #DevTools #Benchmarking
https://arxiv.org/abs/2608.11977
@j-bio-bioprod.bsky.socialKong evaluates #teak-derived quinones (#2-MAQ, lapachol) as #redox-active alternatives to #anthraquinone in #alkaline pulping, #benchmarking them against AQ and identifying gaps in #efficacy, #safety, and #mill compatibility
DOI: 10.1016/j.jobab.2026.100303
www.sciencedirect.com/science/arti...
@perfallit.bsky.socialIf your benchmark wobbles 8% run to run, a 5% fail threshold just blocks random PRs. Measure your noise floor first, then set the gate above it. A threshold below your variance isn't strict, it's a coin flip. #benchmarking #performance #CI
@devstackdaily.bsky.socialA new arXiv benchmark extends CRUXEval with paired short and long traces, testing models on predicting final output and checkpoint state in 400 Python and C++ programs without tools or code execution. Reasoning settings…
#AI #Benchmarking #SoftwareEngineering #LLM
https://arxiv.org/abs/2610.11889
@devstackdaily.bsky.socialNew arXiv work introduces AgenticBBO-Bench, a cross-domain benchmark spanning synthetic functions, hyperparameter optimization, database tuning, chip design, and molecular design to fairly compare LLM-based…
#LLMagents #BlackBoxOptimization #Benchmarking #DevTools
https://arxiv.org/abs/2610.12183
@perfallit.bsky.socialRight, a benchmark check only verifies what you tell it to. Perfall's gate is exactly that and nothing more: you set the allowed percentage, and the CLI exits non-zero only when a change exceeds it. No claim it measures overall system performance. #benchmarking #CI #pytest #golang
@gradientbrief.bsky.socialNew arXiv benchmark OpenProblemBench tests AI on 82 unresolved math and theoretical physics problems, with GPT-6-Astra topping evaluated models at 14.0% mean judged solve rate.
#OpenProblemBench #AIResearch #MachineLearning #Benchmarking
https://arxiv.org/abs/2610.11118
@nimblepros.comWant to benchmark methods in .NET?
⏱️ We have you covered in this step-by-step tutorial: https://bit.ly/3ytwi0x
@perfallit.bsky.socialRight, a gate only enforces what you tell it. The real decision is the percentage you pick. Set a lazy number and any slowdown passes, same as asserting true equals true. The gate is only as meaningful as that threshold. Fail on noise you measured, not a guess. #Benchmarking #CI
@tmlr-pub.bsky.socialcarps: A Framework for Comparing N Hyperparameter Optimizers on M Benchmarks
Carolin Benjamins, Helena Graf, Sarah Segel et al.
Action editor: Kevin Swersky
@informaq-de.bsky.socialTQrouting erreichte 27 neue beste bekannte Lösungen in CVRPLIB-Benchmarks mit Spitzenleistung bei den größten Instanzen. Früher Zugriff ist jetzt für Organisationen verfügbar, die die Auswahlkriterien erfüllen.
@informaq.bsky.socialTQrouting achieved 27 new best-known solutions in CVRPLIB benchmarks with leadership on largest instances. Early access now available for organizations meeting selection criteria.
@devstackdaily.bsky.socialNew arXiv paper ReFract introduces a 150-scenario benchmark for testing whether LLM agents stay within the knowledge and tool boundaries of a user's role in high-stakes settings like industrial maintenance.
#AIagents #LLMs #Benchmarking #SoftwareEngineering
https://arxiv.org/abs/2610.03356
@tiffymarlowe.bsky.socialForty-five minutes of tweaking fan curves just to shave two degrees off the GPU under load. Completely rational use of a Monday afternoon. 🖥️🌡️ #pcbuild #battlestation #benchmarking #desksetup #ai #aigirl
@siliconsignalai.bsky.socialTime series forecasting benchmarks often miss real-world failure modes by relying on Gaussian noise or simple perturbations instead of structured, scenario-grounded stress tests. The rise of TSF…
#TimeSeriesForecasting #Benchmarking #AIInfrastructure #RobustML
https://arxiv.org/abs/2610.02608
@perfallit.bsky.socialNot every slowdown arrives at once. Drift is quieter: each change is a little slower, every one passes review, and nobody can say when it started. Keep every benchmark run tagged with its commit. Old results let you bisect a slowdown the way you bisect a bug. #Benchmarking #CI