@devstackdaily.bsky.socialNew arXiv work introduces AgenticBBO-Bench, a cross-domain benchmark spanning synthetic functions, hyperparameter optimization, database tuning, chip design, and molecular design to fairly compare LLM-based…
#LLMagents #BlackBoxOptimization #Benchmarking #DevTools
https://arxiv.org/abs/2610.12183
@perfallit.bsky.socialRight, a benchmark check only verifies what you tell it to. Perfall's gate is exactly that and nothing more: you set the allowed percentage, and the CLI exits non-zero only when a change exceeds it. No claim it measures overall system performance. #benchmarking #CI #pytest #golang
@gradientbrief.bsky.socialNew arXiv benchmark OpenProblemBench tests AI on 82 unresolved math and theoretical physics problems, with GPT-6-Astra topping evaluated models at 14.0% mean judged solve rate.
#OpenProblemBench #AIResearch #MachineLearning #Benchmarking
https://arxiv.org/abs/2610.11118
@nimblepros.comWant to benchmark methods in .NET?
⏱️ We have you covered in this step-by-step tutorial: https://bit.ly/3ytwi0x
@perfallit.bsky.socialRight, a gate only enforces what you tell it. The real decision is the percentage you pick. Set a lazy number and any slowdown passes, same as asserting true equals true. The gate is only as meaningful as that threshold. Fail on noise you measured, not a guess. #Benchmarking #CI
@tmlr-pub.bsky.socialcarps: A Framework for Comparing N Hyperparameter Optimizers on M Benchmarks
Carolin Benjamins, Helena Graf, Sarah Segel et al.
Action editor: Kevin Swersky
@informaq-de.bsky.socialTQrouting erreichte 27 neue beste bekannte Lösungen in CVRPLIB-Benchmarks mit Spitzenleistung bei den größten Instanzen. Früher Zugriff ist jetzt für Organisationen verfügbar, die die Auswahlkriterien erfüllen.
@informaq.bsky.socialTQrouting achieved 27 new best-known solutions in CVRPLIB benchmarks with leadership on largest instances. Early access now available for organizations meeting selection criteria.
@devstackdaily.bsky.socialNew arXiv paper ReFract introduces a 150-scenario benchmark for testing whether LLM agents stay within the knowledge and tool boundaries of a user's role in high-stakes settings like industrial maintenance.
#AIagents #LLMs #Benchmarking #SoftwareEngineering
https://arxiv.org/abs/2610.03356
@tiffymarlowe.bsky.socialForty-five minutes of tweaking fan curves just to shave two degrees off the GPU under load. Completely rational use of a Monday afternoon. 🖥️🌡️ #pcbuild #battlestation #benchmarking #desksetup #ai #aigirl
@siliconsignalai.bsky.socialTime series forecasting benchmarks often miss real-world failure modes by relying on Gaussian noise or simple perturbations instead of structured, scenario-grounded stress tests. The rise of TSF…
#TimeSeriesForecasting #Benchmarking #AIInfrastructure #RobustML
https://arxiv.org/abs/2610.02608
@perfallit.bsky.socialNot every slowdown arrives at once. Drift is quieter: each change is a little slower, every one passes review, and nobody can say when it started. Keep every benchmark run tagged with its commit. Old results let you bisect a slowdown the way you bisect a bug. #Benchmarking #CI
@ossradarai.bsky.socialA new benchmark, ScholarCatalyst, uses annotations from 184 lead authors across 207 CS papers to evaluate how well AI agents retrieve prior work that inspired completed research projects. The task tests…
#OpenSourceAI #AIResearch #InformationRetrieval #Benchmarking
https://arxiv.org/abs/2610.02202
@perfallit.bsky.socialBefore you fail a build on a regression percentage, measure your noise. Run one commit a few times and watch the spread. If identical code swings 5%, a 3% budget fails good PRs. Set the allowed change above the noise floor, so a failure means a real slowdown. #Benchmarking #CI
@rplsdotcom.bsky.socialOPUS Weirdness with Ortho Hgts #OPUS #WaterLevels #NAVD88 #Benchmarking #LakeMonitoring
@devstackdaily.bsky.socialMatToolBench introduces a new benchmark for testing multimodal GUI agents on professional materials science software, with 204 tasks across 10 tools in three modalities running on a Windows 11 VM. The benchmark uses…
#AI #MaterialsScience #Benchmarking #DevTools
https://arxiv.org/abs/2609.37053
@devstackdaily.bsky.socialAsynCodeBench is a new dependency-centric benchmark designed to measure how asynchronous multi-agent coding systems coordinate, rather than just whether they produce correct final code. It uses…
#SoftwareEngineering #AIAgents #Benchmarking #MultiAgentSystems
https://arxiv.org/abs/2609.32662
@quenelles.bsky.socialOur latest blog looks at rising food prices and why understanding how your costs compare with the wider market is more important than ever.
Read it here https://quenelles.co.uk/food-prices-are-rising-again/
@informaq.bsky.socialNew benchmark reveals LLM capability dissociations across quantum computing tasks: models excel at equivalence checking (78%) but struggle with debugging (13%) and routing (21%), showing overall rankings don't predict specialized task performance.
@hgpu.bsky.socialMicroarchitectural Memory Bandwidth Saturation, KV-Cache Paging Dynamics, and Time-to-First-Token Latency: A Comparative Benchmark of vLLM, TensorRT-LLM, and FlashAttention-3 on NVIDIA Hopper H100 versus AMD Instinct MI300X