AI maturity is fundamentally about expanding the delegation boundary. You start by letting AI assist with code completion, then with features. Eventually, agents write pull requests from requirements with minimal human involvement. Each step hands more responsibility to machines.
But here’s what people skip over: Tests are the primary mechanism for making that delegation safe. You can’t let agents operate autonomously if you can’t verify what they produce. Low test coverage is the single biggest barrier to advancing along the AI maturity curve and most organizations haven’t come to terms with that. They want the AI productivity gains without doing the unglamorous work of building test infrastructure to make those gains trustworthy.
Coding assistants make delivery so fast that most QA organizations are underwater. Teams face a choice nobody wants to make: Attempt ten times the testing work, or test selectively and accept less certainty about whether what’s running is safe. Neither option is sustainable.
It’s time to shift the conversation. The question isn’t how fast teams can ship. It’s whether they can prove, with evidence, that what they shipped works.
What’s Changed
AI accelerated code delivery, but the bigger change is in who’s building software, how much is being built and what it takes to trust it.
People without training in architectures, APIs, or security can now generate functioning applications. That’s a real unlock – and a risk multiplier. A junior developer hard-coding plain-text credentials used to be an isolated incident you’d catch in code review. Now, with AI assistants putting production-grade tools in everyone’s hands, that rookie mistake can show up across dozens of repos before anyone notices.
Throwing more people at the problem is not realistic. Companies have downsized teams. Even if fully staffed, QA has been defined by code-level inspection and automation. Inspection is a tick-box verification of functions, integrations and outputs under predefined conditions. Automation mechanizes that inspection. Neither was designed to keep pace with AI-generated code arriving at this volume.
The other response is to fight AI-generated code with AI-generated tests. Sounds logical. In practice, it creates a different problem because nobody is sure which of those tests matter, which are redundant and what’s missing. Unthoughtful AI-generated tests are just noise.
So if you can’t hire your way out and you can’t generate your way out, what’s left? It’s a governance and intelligence problem. When an agent generates fifty variations of a login test, who decides whether that’s useful or busy work? When AI creates a test suite, who ensures it traces back to requirements? Enterprise teams are demanding answers. They want test cases generated from issue trackers, linked back to originating requirements, organized and traceable, all triggered from within existing workflows.
The organizations that build this layer of trust will scale AI-driven development. The rest will find out the hard way what happens when you ship fast without proof.
From Instructions to Outcomes
QA has historically been about verifying what an application is, not what it does. That approach worked when humans wrote code and applications changed on schedule. It falls apart when AI regenerates parts of the codebase continuously.
The gap between what someone intended the software to do and what it does is where releases go sideways. That gap is widening. Seven of 10 software executives are concerned that application quality is suffering as AI speeds code development, new research shows and just as many are concerned about the impact going forward.
This means the focus on quality needs to catch up to the focus on AI development speed. Product managers and engineers define the requirements. QA must validate that requirements are met. But most QA practices are still oriented around inspecting code artifacts rather than confirming outcomes. The shift needs to be from “does this component work as coded?” to “does this application behave the way the business expects it to?” We call this “Application Integrity,” continuous, measurable assurance that software works as intended with the governance to operate at AI speed and scale.
One pain point I hear from QA teams is about the manual creation of tests, especially for each variation of applications. That’s where agents should be doing the heavy lifting. But AI-driven QA will only help if applied thoughtfully. You need agents that can test based on user behavior, adapt as interfaces change, and prioritize what’s most likely to break rather than what’s easiest to test. The difference between useful AI-driven testing and expensive noise comes down to whether the tools are built with that kind of judgment baked in, or whether they’re just fast.
Maintaining quality when everything is moving this fast requires application integrity, meaning ongoing evidence that the software behaves as intended. Continuous assurance. Can you show, at any given moment, that your software is doing what you said it would? To do that, teams need to get a lot clearer about what they’re measuring. Some of the metrics teams track today are still the right ones. Others need to be added.
- Deployment frequency and lead time for changes aren’t going away. AI should make shipping faster and release cycles shorter. If your quality practice is slowing them down, fix it. Those are throughput metrics. On their own, they don’t tell you whether the things you’re shipping actually work.
- That’s where change failure rate comes in. This is becoming the most important metric. What percentage of deployments cause incidents, rollbacks, or hotfixes? When code volume goes up but verification doesn’t keep pace, this number climbs. Driving this percentage down is the core challenge for quality teams and probably the one with the most direct financial impact.
- Mean time to resolution deserves more attention. If deployments are landing twice as often but recovery time hasn’t improved, your exposure window just doubled.
- Invest in functional coverage: What percentage of your app’s functionality and API surface is covered by tests. And traceability coverage: what percentage of tests and API contracts link back to a requirement or spec? This separates teams who can demonstrate their software works from teams who are pretty sure it does.
A New Standard
Code is being delivered by more people, faster than ever. Now, the new standard is proving the applications work.
from DevOps.com https://ift.tt/oY5LJdO
Comments
Post a Comment