Agents have been one patch after another: retrieval, memory, retries, approvals, tests. Every model upgrade eats some of that scaffolding back. Two things it can't eat: what counts as good, and who decides. So I spend my time on evals. If you can measure it, you can simplify it. #Evals

An arXiv study questions whether higher LLM judge scores on an agentic banking deck harness reflect real document improvements or just shifts in grading criteria. Across five judges and seventeen deliverables, the full system scored 20.4…

#AI #LLMs #FinTech #Evals
https://arxiv.org/abs/2609.39958

New arXiv paper proposes evaluating consumer AI action agents on both trust (no unauthorized actions) and task completion, arguing both depend heavily on the harness around the model rather than the model itself. The evaluation runs…

#consumerAI #AIAgents #evals
https://arxiv.org/abs/2609.33017

RAGWarrant frames RAG deployment as a constrained evidence decision, normalizing evaluator outputs and applying predeclared quality and risk gates before emitting auditable PROMOTE, BLOCK, REJECT, or INCONCLUSIVE verdicts. Worth a look…

#RAG #GenAI #MLOps #Evals
https://arxiv.org/abs/2609.34179

#jev on my #evals test vs Sonnet 5: 10x+ faster, 100x+ cheaper, natural code integration.

Not just an improvement. A new enabler.

For example, we can run many more #llm-judges without the cost eclipsing the product itself.

Tested TypeSafe AI's #Jev for #evals. 29/30 same as Sonnet 5 on my context-awareness dataset — but 12x faster and 164x cheaper.

No more JSON parsing mess. It just gives you the choice and its probability.

Love seeing #specialized #AIs solving specific problems instead of just bigger models.

Do you test your AI agents before launch? 🤖

My colleague Jan-Felix Schmakeit shared 5 rules for evals:
- Match graders to sandboxes
- Write multi-step prompts
- Avoid prompt-grader mismatch
- Grade final outputs, not tool paths
- Curate distinct user datasets

Link👇

#AI #Evals #AgenticAI