AI coding assistants write code fast. Whether that code can be trusted is a separate question, and it’s becoming a more urgent one. Surveys this year put average developer trust in AI-generated output at just above the midpoint of a five-point scale, and more than half of developers admit they ship AI code without testing it first. Coverage numbers on AI-written code often look fine on paper, but plenty of that coverage turns out to be shallow: tests that check a result isn’t null and call it a day.
Microsoft is trying to close that gap with a new open-source agent built specifically for unit testing. Called code-testing-generator, it lives inside the dotnet-test plugin in the dotnet/skills repository, and it’s designed to answer the questions a bare “generate unit tests” prompt leaves open: which code needs coverage, what framework the project already uses, where new tests belong, and whether the build will actually find them.
That last point matters more than it sounds. A new test project can compile and pass locally and still never run in CI, because nobody wired it into the solution or the test command. It’s a quiet failure mode, and one Microsoft says its agent checks for directly.
How it Works
The agent doesn’t jump straight into writing code. It starts by researching the repository: detecting the language, finding the test framework already in use, and studying existing tests to learn local conventions. From there, it scales its approach to the size of the job. A single method gets a direct read-write-validate pass. A larger request triggers planning, and a request to cover a whole subsystem runs an iterative loop until it hits a coverage goal.
Once it starts writing, the agent works from simple code toward code with more dependencies, mapping behaviors to test files as it goes. It runs the tests as it writes them, and if something doesn’t compile or an assertion is wrong, it fixes it before moving on. It won’t touch production code, and it avoids writing tests that hit real URLs, open ports, or depend on exact timing — the kind of shortcuts that make test suites flaky.
The last step is the one most testing tools skip: checking whether the tests are actually useful. Microsoft’s agent runs a lightweight form of mutation testing, introducing small code changes that should break a good test, then confirming the existing tests catch them. It also checks for weak assertions, verifies every requested scenario has a matching test, and runs the full suite before calling the job done.
What the Numbers Show
Microsoft benchmarked the agent against stock GitHub Copilot using the same underlying model, across 152 tasks pulled from real repositories. The specialized agent completed 92.1% of tasks, against 78.9% for stock Copilot — a 63% cut in failures.
The gap was widest on vague prompts, the kind that leave most of the decision-making to the agent. There, the specialized agent passed 88.8% of tasks against 66.3% for stock Copilot. On prompts tied to a specific code diff, the agent passed all 15 test cases in the benchmark; stock Copilot passed none. When prompts were already detailed, the two setups tied, which tracks: a good prompt does the agent’s research for it.
The gain didn’t come from generating more tests. The specialized agent actually produced slightly fewer tests than stock Copilot, with roughly the same average line and branch coverage, and finished about 5.5% faster on average. The improvement was in reliability — getting a working result on the first try — not sheer volume.
Microsoft also ran the comparison across models, including Claude Opus 4.8, GPT-5.5, and Claude Haiku 4.5, and found the workflow helped all three, with the biggest failure reduction on Opus. It held up across languages too, more than doubling the completion rate on Python tasks and passing every Go task in the benchmark, though stock Copilot still edged it out slightly on PowerShell.
A tougher, independent benchmark called SWE Atlas — which checks whether generated tests actually catch injected bugs — showed a smaller but still positive gap: 36.4% task completion for the specialized agent against 27.3% for stock Copilot.
Mitch Ashley, VP and practice lead for software lifecycle engineering and AI-native software engineering at The Futurum Group, says the real story is in the method, not the model. “Coverage numbers have been lying to teams for years. AI-generated code made the lie bigger,” Ashley said. “Microsoft’s agent runs mutation testing on its own output, breaking code to confirm the tests actually notice.”
Ashley points out that the benchmark’s headline number — a 63% cut in failures — traces back to procedure, not a smarter model. “Verification debt does not get paid down by generating more tests,” he said. “Engineering leaders should require proof that a test fails when the code breaks, before coverage targets go into any AI policy.”
The Bigger Pattern
None of this makes AI-written code trustworthy by default. It points at something narrower and more useful: a workflow that treats test generation as a process with checkpoints, not a single prompt-and-done request. Given how much AI-generated code teams are shipping right now, and how thin the testing behind it often is, that kind of structure is worth paying attention to.
The agent is open source and available now through GitHub Copilot CLI, with support in Visual Studio Code and VS Code Insiders in preview. Microsoft says Visual Studio support is coming. It covers a dozen languages beyond .NET, including Python, Go, Java, and Rust, and Microsoft says it’s exploring whether the same approach could extend to other kinds of testing down the line.
from DevOps.com https://ift.tt/uxFYrlc
Comments
Post a Comment