When an end-to-end test fails after a front-end change, the repair often looks routine. A class name changed. A button moved. A selector that used to be unique now matches two elements. A self-healing system inspects the page, proposes a new locator, reruns the test, and gets a green result.
That green result is useful, but it does not prove the test was repaired.
The new locator may point to the wrong button, a hidden duplicate, or an element from another part of the page. The run passes because the automation found something clickable. The test may no longer check the behavior it was written to protect.
I call that a false heal. It is worse than an ordinary failure because it removes the visible warning. A red test creates work. A false heal makes the suite look healthy while weakening its signal.
The practical fix is to treat an AI-generated repair like any other untrusted code change. The healer can propose the patch, but a separate deployment gate must decide whether the patch is safe to merge.
A Passing Rerun Proves Too Little
Most self-healing demonstrations stop at one of two checkpoints: the replacement locator can be inserted, or the test runs without an error. Neither checkpoint answers the question that matters to a release pipeline: does the repaired test still exercise the intended behavior?
Imagine a checkout test that should click the final Submit button. A redesign breaks the original selector. The healer finds another button with similar text, updates the locator, and produces a passing run. If that replacement points to Cancel, a hidden mobile control, or a Submit button from a different form, the execution succeeded for the wrong reason.
The healer cannot grade that repair using the same evidence it used to create it. It needs an external contract that describes what the test is supposed to touch and what outcome must follow.
Put Three Checks Outside the Healer
The first check is target identity. Before applying a repair, record what made the original element the intended target: accessible role, name, stable test identifier, containing form, nearby label, or another product-level constraint. Playwright’s locator guidance favors user-facing attributes such as roles and labels because they carry more meaning than a long CSS path. A replacement should satisfy the same identity constraints rather than merely resolve to one element.
The second check is behavior preservation. Rerun the relevant user path and verify the outcome, not just the click. Did the expected request fire? Did the page reach the right state? Did the original assertion still run? A repaired locator that bypasses or weakens the assertion should fail the gate even if the command exits successfully.
The third check is review scope. The gate should show the original locator, the proposed replacement, the matched element, the diff, and the rerun evidence in one review packet. A human should approve repairs that affect release-critical paths, financial actions, account changes, or security controls. Lower-risk repairs can use a lighter policy, but the evidence should still be retained.
These checks belong outside the model. If the same agent proposes the change, relaxes the assertion, and declares the rerun successful, it can make its own work easier to pass. A deterministic gate prevents the repair process from changing the definition of success.
Record the Decisions CI Usually Forgets
Teams normally keep the final patch and the test result. That is not enough for an AI-assisted repair. The pipeline should also retain the failure that triggered the repair, the original target contract, every proposed locator, the element matched by the accepted locator, the assertions and product outcome observed during the rerun, and the person or policy that approved the change.
Rejected candidates matter. Without them, the audit trail shows only the answer that passed. It cannot explain whether the healer considered several weak matches, whether a confidence threshold was narrowly cleared, or whether a reviewer overrode the recommendation.
This record also makes rollback practical. If the repaired test starts behaving differently after the next UI change, the team can see why the previous repair was accepted instead of reconstructing the decision from a green build and a one-line diff.
Use the Healer as a Candidate Generator
Self-healing automation is still useful. It can collect failure evidence, inspect the current page, rank replacement locators, and prepare a reviewable patch faster than a person starting from an empty ticket.
But candidate generation and release approval are different jobs. The model is good at narrowing the search. The deployment gate is responsible for preserving the test’s meaning.
That division keeps the workflow honest. The agent proposes. Deterministic checks verify identity and behavior. A human reviews the consequential changes. The result may still be a green build, but now the build has evidence that it is green for the right reason.
from DevOps.com https://ift.tt/kPjbval
Comments
Post a Comment