Same pattern this time, and agents now write much of that testing too. So we open-sourced our skills that teach them to write unit tests. Maybe they'll help someone: github.com/mavka-ai/uni...
@maksdan.bsky.social@maksdan.bsky.social
@maksdan.bsky.social
@maksdan.bsky.social@maksdan.bsky.socialGreen tests here mostly prove the agent's own changes work. A silently renamed env var is a case nobody listed to test, and that list is the hard part. Which skills do you use for unit tests?
@maksdan.bsky.socialAgreed, dropping unit tests is a bad take — a study of 198 production failures (OSDI '14) found 77% reproducible with a unit test. The actual problem is agents writing bad ones, not unit tests being useless.
@maksdan.bsky.socialExactly. We use it that way: the LLM writes the tests, deterministic mutations judge them. Every agent-written suite in our benchmark had 100% coverage, but they caught anywhere from 5 to 11 of 12 planted bugs: github.com/mavka-ai/uni...
@maksdan.bsky.socialAgree. We measured it: agent-written tests at 100% branch coverage caught only 5 of 12 planted bugs in Spring Petclinic. Coverage said all fine, mutation testing showed the gap: github.com/mavka-ai/uni...