Mutation testing: can you trust AI-generated tests?

Coverage lost its meaning the moment it became free. An agent adds two hundred cases in a minute. Coverage jumps to ninety percent, CI goes green. None of those three things answers the only question worth asking: if this code were wrong, would any assertion break. Mutation testing answers it mechanically. It is the one measurement of test quality that did not get cheaper when writing tests did.

How mutation testing works

Mutation testing is a technique for judging the quality of a test suite: a tool makes one small, deliberate change to production code and checks whether any test catches it. You take one production file and change one thing in it. Greater-or-equal becomes greater. A conjunction becomes a disjunction. A condition becomes true. A loop step runs the other way. A whole branch disappears. The result is a mutant. You run the tests that were supposed to guard that line.

• A test failed, so the mutant is killed. Your tests really do enforce that decision.
• Everything passed green with broken code, so the mutant survived. That is a finding. You have a line your tests execute and say nothing about.

Repeat a few hundred times per file. The mutation score is the share of killed mutants among the ones that could be judged at all.

The whole idea fits in one sentence. A test checks your code. A mutation test checks your test.

The idea is half a century old and rests on two assumptions. Both are worth knowing before you believe any number. First: programmers write nearly correct code, so real bugs are small deviations from a good version rather than random noise. Second: a suite that catches those small, single-point faults also catches the compound ones. What is new is only that the technique finally pays off, and the reason has nothing to do with the technique.

What coverage will not tell you

Coverage tells you whether a line ran. Mutation tells you whether anyone asserts anything about it. Two different quantities, and only the second one matters.

A test that calls a function and never checks the result gives you one hundred percent coverage. A test that stubs every dependency and asserts on the stub gives you one hundred percent coverage. A test that recorded the current output as the expected one gives you one hundred percent coverage. That last one stays green when the output is wrong.

This is not a niche observation. In April 2026 Thoughtworks put mutation testing into the Trial ring of its Technology Radar for the first time. The stated reason is exactly this: high coverage percentages can mask logically hollow tests or generated code that has never been meaningfully asserted.

AI in software testing: what changes when a model writes the tests

Four things at once, and only the first is obvious.

Writing tests got cheap, judging them did not. Two hundred generated cases land in a pull request. The same person reads them as before, only now there is ten times more to read. So they stop reading. Mutation score keeps pace with how fast tests now appear, because a machine computes it.

The failure mode of a model-written test is precisely the one mutation catches. A model writing a test against finished code has that code in context. The cheapest route to green runs through a test that repeats the implementation. It stubs the dependencies and asserts on the stub. It copies the current result into the expected one. It checks that something got called instead of checking what came out. Such a test executes, passes and raises coverage. It would also pass with the condition inverted. That is the definition of a mutant that survived.

This is not an opinion. A team in Luxembourg, the same group that has worked on mutation for years, went through twenty-four projects. Models write assertions that describe actual behaviour rather than expected behaviour. Random generators had the same problem before them. When the model judges for itself whether an assertion is right, it lands somewhere in the forties. On buggy code it falls another eight to nine points. So the model is strong as an author and weak as a reviewer, which is the same trap behind why a model cannot be its own reviewer.

You can see it in what agents actually commit. A study from this year went through more than 1.2 million commits from 2025 across more than two thousand repositories. Of the agent commits that touch test files at all, thirty-six percent add mocks. For the same kind of commit written without an agent, twenty-six. The authors say it plainly: put your mocking rules in the agent's config file.

When an agent can edit the tests, the tests stop being an independent measure. The task reads: make CI green. Weakening an assertion satisfies that task and is a legal move. A benchmark published late last year measured which route models take to a green run. Editing the test file itself turned out to be rare: under one percent on well-specified problems, and zero for all three agents on the ones whose oracle was arbitrary. One of the three did delete the test file outright, in three percent of the well-specified runs. Something else rises instead. Satisfying the visible test cases without solving the problem jumps from about two percent to twenty-two, and to forty-four for the worst agent. That is nine problems and the authors call the sample noisy, but the direction is not in doubt. A different route, the same lesson: a check the agent can see is a check it can satisfy without solving anything. So it is worth settling in advance whether the test file belongs to what an agent changes without asking.

Mutation score has a property here that coverage lacks. Delete an assertion and coverage stays exactly where it was. Mutation score drops. A measure has to sit beyond the reach of whatever it measures, which is the same reason to iterate an agent only against a measure it cannot move itself.

A growing share of your code exists to police the model. Trust rules, response validation, parsing, fallbacks, retries, limits. That code runs on the rare path. Tests get written for the common one. So it is exactly the part of the system a green suite says nothing about.

What it found in production code

That is as much context as I can give. A production B2B application, where a model extracts values out of inbound documents and the code around it decides whether to trust them. The suite: over a thousand examples, zero failures, green for a long time.

I wrote the harness from scratch. The off-the-shelf one failed on three independent counts at once. It needs a newer version of the language than the app runs on. It bumps a parser library shared with the linter, so a testing tool would cost us a diff across every lint rule. And it is commercially licensed for private repositories. Sixteen operators, chosen for the bug classes that cost the most in this kind of code. Boundaries and off-by-one. Swallowed exceptions. In-place mutation of shared state.

First audit of one file: 486 mutants. 235 killed, 204 survived, 44 on lines no selected test executes. I am holding back the percentage, because the percentage is not the finding. Three things came out of it, and only one of them reads "write a test".

A real bug

One function decides which value to accept: the one from the model, or the one extracted from the document. It had two trust rules and both looked only at the shape and length of the text. Neither asked whether that text had anything to do with the place it supposedly came from. A short, well-formed, entirely invented value won. Nothing downstream re-checked it. The neighbouring test avoided that path only because its two strings happened to be the same length.

That is a finding about the design, not about the tests. The trust rule checked the format of a value and never its provenance, and those are two different things. The guard existed, somebody wrote it, and it passed review. Nobody ever checked it, which is exactly what is meant by a failure that lights no lamp. The fix holds to one rule. If a value shares no token with the source text our own extraction reconstructs, it cannot have come from that source. I checked the rule against the full historical set, not a sample.

Dead code

Across the whole first audit, no longer just that one file, 106 mutants sat on lines no test executes. All of them in four methods. None of those four methods has a caller anywhere in the repository. They arrived together in a single import and nobody touched them since. Result: 214 deleted lines and one decision instead of 106 tickets.

Fresh code with fresh tests

The third one is the most uncomfortable. I ran the harness on its own operators file. Hours earlier that file had gained four new operators and 101 lines of new tests, in the same commit. 97 mutants, 87 killed, 90.6 percent. Five of the nine survivors sat in code written that same day, alongside those tests. One exposed a dead null guard where null cannot occur. Another hit a condition no example ever reached.

Ninety percent is a good score. Nine unguarded decisions in a file that had just passed full review and received a fresh set of tests is also a fact.

The mutation score lies without an "I don't know" bucket

This is where measurement parts ways with a dashboard. This is the part that gets skipped most often.

My harness distinguishes eight outcomes per mutant. Only two of them say anything about your tests: killed and survived. The other six are the harness admitting it has no verdict. The mutant hung. The mutant blew up the process. No selected test executes that line. The file cannot be audited at all. The mutant did not parse. The child process died before the harness installed the mutant.

Put those six in the denominator and you get a higher, useless number, which is one more figure that looks like evidence and is not. Separate them and you get something you can work with. On one file, 47 of 54 entries that looked like survivors turned out to be lines nobody executes, in three methods nobody calls. Glued together they read as 54 tickets and a score of 27 percent. Separated they read as three findings and a score of 74 percent of what the tests actually reach. The second version is true. The first sounds scarier.

The same split separates two different jobs. A surviving mutant is a test that executes something and stays silent about it, so the fix is a stronger assertion in a test that already exists. A mutant on an uncovered line means there is no test, so the fix is a test that does not exist yet. Two different jobs, two different budgets.

And one thing without which the rest is guesswork. Before you count anything, run the tests against the unmutated file. If that is not green, no number for that file means anything. Here, that step caught a file that resets its own mutex when reloaded. Without the check the report would have shown a wall of killed mutants and a beautiful score. A reload side effect would have killed every one of them, and no test would have killed any.

When a model stands on both sides, that stops being hygiene. It becomes the thing every number after it rests on. An agent that writes the code, the tests and the mutants has closed the loop around itself. The measure survives that arrangement only if it can say "I don't know". And only if it refuses to credit a defence it cannot prove.

How to run mutation testing without killing CI

Mutation is expensive by construction, because one mutant means one test run. A few hundred mutants per file times your suite's startup time turns a coffee break into an overnight job. Google has been doing this for over a decade and published the bill. Almost 17 million mutants generated across 776,000 changes. 2.1 million of them shown to humans in code review, across more than 24,000 developers.

The most important number in that bill has nothing to do with performance. At the start, developers judged 85 percent of the mutants shown to them as useless. It took six years of collected feedback and rules that suppress whole classes of mutants to lift the productive share from 15 to 89 percent. The median number of mutants per change fell from 820 to seven along the way.

The conclusion is inconvenient for anyone who just wants to switch a tool on. A raw audit produces mostly noise, and that noise costs the time of the most expensive people in the company. Four things bring the cost back to something sane.

• Mutate the diff, not the repository. The only mode that survives contact with a team is "only the lines this branch changed". The agent version is simpler: mutate what the model just wrote. Google measured what that costs: on medium and large changes, commit-level mutation slows review no more than computing coverage does, and a visible overhead shows up only on the smallest changes.
• Run only the tests that could kill. A mutant on a line a given test file never executes is wasted budget. Line-level coverage tells you which tests stand a chance.
• Start the process once. A fresh suite start per mutant costs me 1.98 seconds for framework-free tests and 17.24 seconds with a full application boot. Of those 17.24, fully 16.15 is loading that does not depend on which mutant is about to run. A fork of an already-booted session costs about half a second.
• Pick the files by hand. I declare the list of audited files and the list of tests meant to be their coverage explicitly. Naming conventions do not work: in a real codebase some tests are named after a scenario rather than a class, and one will cover a file no naming rule would ever associate with it.

The tooling is ready and mature. Java has PIT. JavaScript and TypeScript have Stryker, C# has Stryker.NET. PHP has Infection. Python has mutmut and cosmic-ray. Rust has cargo-mutants. Ruby has the mutant gem. Before you pick one, check the two things I tripped over: the minimum language version and the licence for a private repository.

What mutation testing will not catch

It will not replace tests you never wrote. At twenty percent coverage, mutation tells you what you already know, only at a higher price. It is an instrument for judging tests that exist.

It does not reach one hundred percent, and that is not the goal. Some survivors are not findings at all. The mutated code is genuinely equivalent to the original and no test could tell them apart. Two shapes recur. A comparison whose operands are already an epsilon apart. A redundant early return whose fall-through computes the same answer. Equivalence is undecidable in general, so a nonzero survivor count is normal even with excellent tests. Google’s team put it plainly in an earlier paper: full mutation adequacy is neither practical nor desirable.

Mutation score is a terrible target. Set a threshold in CI and you get exactly what you asked for and nothing else: tests written to kill mutants. Treat the number as a thermometer. The work is in the survivor list read by method, not in a bar on a dashboard.

Not every "defended" means defended. In my harness a mutant that hangs or blows up the process counts as defended. Broken code that hangs or explodes will not reach production either. But neither outcome proves that any assertion stated anything. A file with many of those has an inflated score and the report has to show it.

Not everything can be measured. My harness refuses ActiveRecord models by name. It installs a mutant by re-evaluating the whole file, and on a model that does not replace the definitions, it appends them. Validations go 9, then 18, then 27. The unmutated baseline run doubles the same way, so it stays green. Then a duplicated validation message kills every mutant, and no test kills any. A wall of false kills and a great score. Refusing is the only honest answer there.

And the most important limit, the one the AI pieces leave out. Mutation is a unit-level instrument. The most common failure in agent-written code does not sit in one line. It sits in two correct pieces that do not compose. Mutation will not see that, and is not meant to.

Where to start

Take one file. Preferably the one you are afraid to touch, and the one holding your trust rules for the model. Run mutation on it. Ignore the percentage and read the survivor list grouped by method. The densest group is one story and one test to write, not twenty tickets.

Then wire it into the diff. Exactly where the agent hands work back.

Frequently asked questions