Self-healing tests: what AI should fix and what it must never touch

Self-healing tests are automated tests that get repaired when the product changes in ways unrelated to what they check: a button gets a new id, a label is renamed, a flow gains a step. Either a runtime healer swaps a broken locator for the closest match while the test runs, or an AI agent reads the failure afterwards and proposes a change to the test code. Done well, self-healing test automation takes routine upkeep off your team every time the interface moves.

Done badly, it turns red tests green by changing what they check, and a real bug ships with a passing suite. Our rule: a fix may change how a test reaches the thing it checks, never what it checks, unless a person agrees. Below are both kinds of healing, the safe changes and the risky ones, an example, and a review workflow that keeps AI test maintenance honest.

What self-healing tests actually do: two mechanisms

The same word covers two mechanisms that behave very differently.

Runtime healing: another element, chosen during the run

The test code does not change. When a locator finds nothing, the tool picks the element that best matches what it recorded on earlier passing runs. Healenium, an open-source library for Selenium, documents the mechanism: it stores the locator from a successful run, catches the “element not found” error, compares the current page with the stored path, takes the candidate with the highest score and carries on, then reports the healed locators after the run.

Agent healing: a patch proposed after the run

An AI agent reads the failure, replays it, inspects the current page and edits the test file. Nothing is decided during the run; the result is a diff someone can read before it is merged.

For self-healing tests in Playwright, this is the official route. Playwright ships three test agents (planner, generator and healer), added in version 1.56 in October 2025 and still there in 1.64.1, the current release as of October 18, 2026. Per the docs, when a test fails the healer replays the failing steps, inspects the current UI to locate equivalent elements or flows, suggests a patch (a locator update, a wait adjustment, a data fix) and reruns the test “until it passes or until guardrails stop the loop.” Its output: “A passing test, or a skipped test if the healer believes that functionality is broken.” npx playwright init-agents adds the agent definitions to a project, and the docs say to regenerate them whenever Playwright is updated.

The agent definitions are worth reading. In version 1.64.1, the healer’s instructions include “Fixing assertions and expected values” next to “Updating selectors to match current application state.” It is told not to ask the user questions but to “do the most reasonable thing possible to pass the test,” and, when the error persists and it is confident the test is correct, to mark the test test.fixme() (skipped) with a comment. None of this is hidden, and a commented skip is an honest signal. But the healer’s patches can contain exactly the changes that need a person, so review them like any other code.

Aspect	Runtime healing	Agent healing
When it decides	During the run	After the run
What changes	The element used in that run	The test code
What you review	A report, if someone reads it	A diff, before it is merged

Why runtime healing can hide real bugs

A locator fails for one of two reasons: the test is out of date, or the product broke. A runtime healer decides from the page and its records of earlier runs, but the evidence that tells the two apart (the ticket, the design, the intent behind the change) is not on the page. Two examples:

• The failure was an accessibility bug. A refactor turns the checkout button into an icon with no accessible name. A locator by role and name rightly fails: role locators follow how assistive technology perceives the page, and the button has lost its name. A healer that matches on position or CSS class finds it anyway, and the run passes.
• A neighbor stands in for an element that is gone. The total line disappears from the order summary. The closest match is the subtotal, which shows the same amount because the test cart has no shipping cost. The assertion passes, and the total is missing in production.

Either way, the build passed before anyone read the report. We would rather the run fail and a person decide in a pull request.

What a fix may safely change

Safe changes alter how the test reaches the thing it checks and leave the check alone.

A locator that moved

Same element, same job, found another way: a new id, wrapper or position. Update the locator, ideally to one that survives the next redesign. Playwright’s docs recommend role locators as “the closest way to how users and assistive technology perceive the page,” and warn that CSS classes can change when a designer changes something.

A label that was renamed on purpose

“Postal code” became “ZIP code,” and the ticket or the design says so. If the label only locates the field, update the locator. If the test checks the label itself, it is an expected value (next section).

A new step in a flow

Checkout gained an address confirmation page. The fix adds the step and changes no assertion, as long as the step belongs to the intended flow. A step that closes an error dialog is a bug report, not maintenance.

A wait that should be an assertion

A fixed sleep such as page.waitForTimeout(3000) becomes a web first assertion that waits for the real condition, such as a heading becoming visible. Playwright marks waitForTimeout as discouraged: “Tests that wait for time are inherently flaky.” The assertion waits only until the heading appears (up to its timeout) and fails if it never does, so the test gets stricter. Our post on flaky tests in Playwright covers timing in depth.

What a fix must never change without a person

Each of these turns a failing test green without fixing the product. An agent may suggest one, with its reasons; a person decides.

• The strength of an assertion. toHaveText checks that the element has the given text, toContainText that it contains it, and toBeVisible only that the element is attached and visible. Every step down that scale proves less.
• An expected value. A total, an error message, a status code. Setting it to whatever the page shows now makes the test assert the bug. A new value needs a source: the ticket, the spec or the person who owns the rule.
• A removed check or a removed test. Even when a feature is really gone, removing its test is a decision about coverage.
• A skipped test. Playwright does not run a test marked test.skip() or test.fixme(), so the flow loses its protection until someone remembers it. A known bug is better pinned with test.fail(): Playwright still runs the test and complains if it unexpectedly passes.
• A raised timeout or more retries. The defaults are 30 seconds per test and 5 seconds per assertion. Raising them hides slowness your users will feel, and retries turn an intermittent failure into a pass (Playwright labels such a test “flaky,” which is worth reading).
• A check moved where it cannot fail. Inside a try whose catch swallows the error, behind an if, after an early return.
• Test data moved off its boundary. A free shipping test uses carts just below and just above the threshold. Nudging the price until the test passes removes the reason it exists.

A before/after example in Playwright

Picture one release with two changes. A redesign removes the checkout-btn id from the checkout button. The same release brings a bug that charges shipping twice, so the order total reads $59.00 instead of $54.00. The test below fails on both.

// pages/cart-page.ts
import type { Locator, Page } from '@playwright/test';

export class CartPage {
  readonly checkoutButton: Locator;
  readonly orderTotal: Locator;

  constructor(page: Page) {
    this.checkoutButton = page.locator('#checkout-btn');
    this.orderTotal = page.getByTestId('order-total');
  }
}

// tests/cart.spec.ts
import { test, expect } from '@playwright/test';
import { CartPage } from '../pages/cart-page';

test('the order total includes shipping', async ({ page }) => {
  const cart = new CartPage(page);
  await page.goto('/cart');
  await cart.checkoutButton.click();
  await expect(cart.orderTotal).toHaveText('$54.00');
});

The legitimate fix touches one line: same button, same job, now found the way a user finds it.

// pages/cart-page.ts: the only line that changes
this.checkoutButton = page.getByRole('button', { name: 'Check out' });

The “fix” that weakens the assertion makes the run green too:

// tests/cart.spec.ts, before
await expect(cart.orderTotal).toHaveText('$54.00');

// after: passes whatever the total says, double shipping included
await expect(cart.orderTotal).toBeVisible();

A quieter version keeps toHaveText and sets the expected value to '$59.00', so the test now asserts the bug. Only someone who knows whether the shipping price was meant to change can approve that.

Both edits can arrive in one tidy diff. We ran them through the heal guard in our sample project (described below), with the files in its folders. It allows the locator change as maintenance, refuses the toBeVisible change as a weakened assertion even when evidence is offered, and refuses '$59.00' without evidence that the new value was intended.

A review workflow for self-healing tests

Five steps keep self-healing honest, whoever or whatever proposes the fix.

1. Classify the failure before anyone edits the test

A red test is evidence about the product until shown otherwise. Put each failure in one bucket: product bug, intended change, test defect (a wrong locator, wait or data), environment problem, flaky test, or unknown. Only intended changes and test defects get a test fix. A product bug gets a bug report, and its test stays red or is pinned with test.fail() and the bug’s id. An unknown is a question for whoever owns the product.

2. Every fix is a pull request

No agent commits to the main branch. Each fix arrives as a pull request with the failure, its class and the evidence: the error, the trace or a screenshot, and for an intended change, the ticket that states the new behavior. Small pull requests, grouped by cause, are easier to judge than one big “fix all tests” change.

3. An automatic check refuses fixes that weaken a test

Reviewers get tired, and a weakened assertion is one line in a long diff, so a script checks every test change first. In our open-source sample project, the heal guard (qa/heal-guard.ts) parses tests and framework code with the TypeScript compiler, compares every assertion with the base branch (its subject, matcher, expected value and what surrounds it) and sorts each change:

• Always refused: a removed assertion or test; a weaker matcher (exact value, then partial, then presence); an assertion moved inside a try that swallows errors, a .catch(), a condition or after a return; any new skip, fixme, hard wait, timeout, retry, configure() or toPass(); any change to the Playwright config or the timeouts file.
• Allowed only with evidence: a new expected value, changed test data or a changed API schema. Evidence is a product document (a rule, feature, card or overview) that states the new behavior, is already on the base branch, is untouched by the diff and changed after the test did, or a person’s quoted confirmation. Application code never counts, “since a code change is also what a bug looks like.”
• Allowed and listed: maintenance, such as a locator in a page object, a helper or a renamed property.

It is strict on purpose: in a spec file, any literal that changes or disappears counts as test data, so even replacing a hard wait there needs evidence (one more reason to keep locators in page objects). It runs in the agent’s job, again on the patch before the pull request opens, and in CI. Its documentation names its limit: it reads the syntax, not what runs, so an assertion moved into a callback that is never called gets past it. A guard is a floor under review, not a replacement.

4. A person approves each one

Before merging, the reviewer confirms that:

• The classification matches the evidence: the trace, the screenshot, the ticket.
• The diff changes how the test reaches things, not what it checks.
• Every changed expected value cites the ticket or document that states it.
• The test fails for the right reason before the fix and passes after it.

If the reviewer cannot tell, the answer is no, and the failure goes to the feature’s owner as a question.

5. Keep a log

One entry per fix, in the pull request description or a file next to the tests:

## Test fix: <date>

- Test: <file and title>
- Failure: <error, link to the trace>
- Class: test defect | intended change
- Evidence: <ticket, design or document that states the new behavior>
- Change: <what moved: locator, step, wait>
- Guard: <OK, or what it refused>
- Approved by: <name>, in <link to the pull request>

The log turns single fixes into patterns: the same page breaking every week points to a locator strategy problem, and a test healed three times in a month needs a closer look. When a bug slips through, the log shows whether a fix hid it.

How we do it

This is how test maintenance works in our service. Claude agents run the suites, collect evidence for each failure and propose fixes as pull requests in your repository. An automatic check rejects any fix that would weaken what a test asserts. A QA engineer looks at every failure before it becomes a bug report or a test fix, and reviews every change before it is merged. The agentic QA page describes the service, and our post on agentic testing with Playwright covers the rest of the agents’ work.

Sources

• Playwright Test Agents, playwright.dev
• Healer agent definition, version 1.64.1, microsoft/playwright on GitHub
• Playwright 1.56.0 and 1.64.1 release notes, microsoft/playwright on GitHub
• Locators and Best practices, playwright.dev
• Locator assertions, playwright.dev
• The page.waitForTimeout() reference, playwright.dev
• Annotations, Timeouts and Retries, playwright.dev
• Healenium overview and How Healenium works, healenium.io

Researched and drafted with Claude, Anthropic's AI model, and checked against the sources it links. Spotted an error? Tell us and we will correct it.