Risk-based testing: decide what to test first with a simple matrix

Risk-based testing means deciding what to test, how deeply and in what order by asking two questions about each part of your product: how likely is it to break, and how much would it hurt if it did? For a small team with no QA department, it is the practical answer to “what do we test first?” You will never test everything, so you choose on purpose what runs on every pull request, what gets a quick check before a release, and what you knowingly leave untested.

The whole risk-based testing strategy fits on one page. List your user flows, score each one from 1 to 3 for likelihood and for impact, multiply the two, and let the score set the test depth: end-to-end tests on every pull request, a smoke check before each release, exploratory testing only, or not tested yet and written down as a risk.

What risk-based testing means

The ISTQB glossary defines risk-based testing as “a test approach in which the management, selection, prioritization, and use of test activities and resources are based on corresponding risk types and risk levels” (ISTQB Glossary, version 3 of the term, checked October 18, 2026). A risk level is “the measure of a risk defined by risk impact and risk likelihood”. In the same glossary, likelihood is the probability that a risk becomes an actual event, and impact is the damage it causes when it does.

The ISTQB Foundation Level syllabus (version 4.0.1, the edition on the ISTQB certification page as of October 2026) gives the reason: testing everything is not feasible except in trivial cases, so it recommends test techniques, test case prioritization and risk-based testing to focus the effort. It describes two ways to assess a risk: in the quantitative approach the risk level is likelihood multiplied by impact, and in the qualitative approach it is read from a risk matrix. The method below mixes the two, which the syllabus allows: scores small enough for a 3 by 3 grid, multiplied into one number you can sort by.

Step 1: list your user flows

Score user flows, not features or code modules. A user flow is one journey through the product from start to finish, such as sign-up or checkout. It is what users notice when something breaks, and what an end-to-end test checks.

To build the list, walk through the product as a new user, then as a paying one, and add what you find in:

• the main navigation and the pricing page
• the support inbox and the bug tracker: which flows do people write in about?
• your incident notes: what broke in production recently?
• anything that touches money, personal data or permissions

Name each flow by what the user does (“change the plan”, “export all projects”). List a flow once per platform when the platforms build it differently: if the web app takes cards through your payment provider and the iOS app sells through the App Store, those are two code paths and two risks.

Step 2: score likelihood and impact from 1 to 3

Three levels are easy to agree on. A scale of 1 to 10 invites arguments about a 6 versus a 7 that nobody can settle with evidence.

Likelihood: how likely is the flow to break?

• 1, low: the code rarely changes, the logic is simple, and the flow has no history of bugs.
• 2, medium: it changes now and then, or it has some logic or an integration that can fail.
• 3, high: it changes often or changes in this release, has complex rules (money, dates, permissions), depends on a third-party service, runs on several platforms, or has broken before.

Past bugs count. “Defects cluster together” is one of the testing principles in the ISTQB syllabus: a small number of components usually contain most of the defects, and the clusters already seen in testing or in production are an important input for risk-based testing.

Impact: how much does it hurt if it breaks?

• 1, low: an annoyance with a workaround, noticed by few users.
• 2, medium: something some users need is blocked and support tickets arrive, but no money or data is lost.
• 3, high: users cannot get in or cannot pay, money is charged wrongly, data is lost or exposed, or every user is affected.

Let the right people score: engineers who know the code judge likelihood best, and people close to customers (a founder, support, sales) judge impact best. Score together in one short session, and when two people are a full point apart, write down the reason instead of averaging.

A risk-based testing example: a subscription app on web and mobile

Here is a matrix for a typical subscription product: a web app, iOS and Android apps, and billing through a payment provider. The scores are an illustration; yours will differ.

Flow	Likelihood × impact	Score
Checkout (upgrade to a paid plan)	3 × 3	9
Sign-up	3 × 3	9
Login	2 × 3	6
Billing change (plan, card, cancel)	2 × 3	6
Search	2 × 2	4
Data export	1 × 3	3
Settings (profile, notifications)	2 × 1	2
Keyboard shortcuts	1 × 1	1

Why each flow scored what it did:

• Checkout depends on a payment provider, plans and prices change, and the web and mobile payment paths differ. When it fails, revenue stops.
• Sign-up changes with every marketing experiment and sends a verification email. A broken sign-up loses every new user, and existing users never notice.
• Login rarely changes, but when it fails nobody gets in.
• Billing change is used less often than checkout, but a wrong charge costs money and trust.
• Search breaks when the index or the ranking changes, but users can still browse.
• Data export almost never changes, but customers trust the file to be complete, and a silent loss of rows is hard to undo.
• Settings change with most redesigns, and a broken profile form is an annoyance.
• Keyboard shortcuts rarely change, and the mouse is a workaround.

Sorted by score, the list is also your test order. The ISTQB syllabus calls this risk-based prioritization: the test cases that cover the most important risks run first. On a tie, put the higher impact first, so a 2 × 3 goes before a 3 × 2.

Keep the matrix in the repository, next to the tests, so a new score goes through a pull request and the history shows why a flow moved.

From score to test depth

The score sets how deeply and how often each flow is tested. Four tiers are enough:

Score	What we do
6 or 9	End-to-end tests on every pull request
3 or 4	A smoke check before each release
2	Exploratory testing when the flow changes
1	Not tested yet, listed as a risk

The syllabus describes this use of a product risk analysis: it may influence the thoroughness and the scope of testing.

6 or 9: end-to-end on every pull request

These tests cover the main path of the flow and the failure most likely to happen (an expired card, a wrong password), on every platform you ship. They block merges, so they have to be fast and stable: a flaky test in this tier teaches the team to ignore a red build. A 9 also gets exploratory testing by a person whenever the flow changes. The syllabus explains why: tests repeated many times become less effective at finding new defects, although repeating them can still pay off in automated regression testing.

3 or 4: a smoke check before each release

One pass through the main path on the release build, automated if the test exists and by hand if it does not. Minutes, not hours. Note where the cut sits: the lowest score a high impact flow can get is 1 × 3 = 3, so a flow that would hurt badly never falls below a smoke check, however rarely it breaks.

2: exploratory testing when the flow changes

No automated test yet. When a pull request touches the flow, someone tries it for a few minutes with a goal in mind (long names, emoji, an empty field, a slow network) and notes what they saw.

1: not tested yet, and written down

Leaving a flow untested is a valid decision when it is visible. The syllabus lists risk acceptance next to mitigation by testing among the possible responses to a risk. Put these flows in the release report, so the person who makes the Go/No-Go call knows what was not tested.

Wire the tiers into CI with tags

Playwright Test can tag a test or a whole group with a tag that starts with @, and then run only the tagged tests with --grep (Playwright docs). Tag by tier, not by score, so moving a flow to another tier changes a single line:

import { test } from '@playwright/test';

// Checkout scored 9: runs on every pull request.
test.describe('checkout', { tag: '@pr' }, () => {
  test('upgrade from the free plan with a card', async ({ page }) => {
    // ...
  });
});

// Search scored 4: runs in the smoke check before each release.
test('search finds a project by name', { tag: '@smoke' }, async ({ page }) => {
  // ...
});

# Every pull request: the flows that scored 6 or 9
npx playwright test --grep @pr

# Before a release: the smoke check plus the pull request tier
npx playwright test --grep "@pr|@smoke"

Maestro flows for iOS and Android work the same way: a tags list at the top of each flow file, and --include-tags on the command line (Maestro docs).

When to rescore

A matrix is a snapshot of the product on the day you scored it. Rescore a flow when:

• A new feature lands or a flow is rebuilt. New code has no track record, so likelihood goes up. If search moves to a new engine, it goes from 2 × 2 to 3 × 2 and joins the pull request tier.
• An incident happens. A bug that reached users is new evidence about likelihood, impact or both, and the syllabus lists reprioritizing tests when an identified risk becomes an issue among the actions of test control. If an export silently dropped rows, data export goes from 1 × 3 to 2 × 3, a score of 6.
• A flow breaks again and again. Repeated failures in CI or in production mean a likelihood of 3, whatever the first score said.
• The business changes. A new pricing model, a large customer who depends on exports, or a move into a market with stricter rules raises impact.
• A flow stays quiet. After a long stretch without changes or bugs, likelihood can come down. Impact comes down only when the business changes.

Between those triggers, a 15-minute review at the start of each release cycle keeps the scores honest.

Common mistakes with a risk matrix

• Scoring modules instead of flows. “The billing service” cannot be tested end-to-end. “Change the plan” can.
• Everything is a 9. When most flows score 9, the scale has stopped sorting. Ask which one you would fix first if two broke at once, and score again.
• One person scores alone. The engineer who wrote checkout may underrate its likelihood, and the founder who sold a large contract may overrate the export. Mix the views.
• Hiding what is not tested. An untested flow nobody wrote down is a surprise waiting to happen. A listed one is a decision.
• Counting tests instead of risks. Many tests on settings and none on billing change look like good coverage and protect little.
• Moving the cuts without checking impact 3. If you change the tiers, make sure a 1 × 3 still gets at least a smoke check. A rare failure that loses customer data still loses customer data.

How we rank flows in week 1

This is how our pilot is planned to start. In week 1 we set up access, rank your user flows by risk with you, agree the success criteria in writing and get the first tests running in your CI. In weeks 2 to 4, the flows that would hurt most if they broke run on every pull request, and a person reviews every failure. Claude agents draft the plan from your specs, tickets and code and write the tests, which live in your repository. A human QA engineer sets the priorities with you, reviews every change before it is merged and signs the Go or No-Go call before each release, with the open risks listed. Shipping stays your decision. The how it works page shows the first weeks in order, and our open-source sample project includes a risk-based test plan for a demo shop.

Sources

• Risk-based testing, ISTQB Glossary
• Risk level, ISTQB Glossary
• Certified Tester Foundation Level Syllabus v4.0.1 (PDF), ISTQB
• Annotations: tag tests, Playwright documentation
• Test discovery and tags, Maestro documentation

Researched and drafted with Claude, Anthropic's AI model, and checked against the sources it links. Spotted an error? Tell us and we will correct it.