Commit-Triggered Test Plans in Practice: A 90-Day Field Report
I run security and deployment at StudioX, which means my job is to be the person in the room asking "yes, but what actually happens on Tuesday morning?" Architecture diagrams are lovely. I care about what a real team does with them once the Mission is live inside their perimeter and the demo glow has worn off. So let me tell you about a team I'll call the Checkout squad, and what changed for them when the Commit-Triggered Test Plan went into production.
If you want the leadership framing, my colleague Mark covers why this matters, and the how it works piece has the full architecture. This is the field report.
Tuesday, before
The Checkout squad ships fast — roughly 40 pull requests a week across payments, coupons, and tax. Their QA lead, Priya, used to spend the first two hours of every day the same way: triaging the overnight PR queue, opening each diff, and reconstructing what the engineer meant. A 340-line coupon refactor with a two-word description would eat 40 minutes on its own before she wrote a single test case. Multiply that across the queue and QA was, on a busy week, spending the better part of a full engineer's time just re-deriving intent that had existed an hour earlier in the committer's head.
The cost wasn't only hours. Two months before we deployed, a coupon change interacted with tax rounding in a way nobody's hand-built plan caught. It escaped to production and refunded a few thousand euros more than it should have before anyone noticed. Not a catastrophe — but exactly the kind of second-order edge case that dies in a cold-start test plan written under deadline pressure.
Tuesday, after
Now the sequence looks different. An engineer commits the coupon refactor at 8:52 a.m. Before Priya has finished her coffee, the Mission has already run: the Change Analysis Agent read the diff and flagged the coupon-lookup and tax-application flows; the Coverage Agent matched 12 of the 200 checkout regression cases as relevant and found two promo-stacking edge cases with no existing coverage; the Risk Ranking Agent ordered the twelve by blast radius and pushed the two uncovered edge cases to the top with a note. All of it read-only — nothing touched a production system.
When Priya opens the portal at 9:05, she isn't staring at a raw diff. She's reacting to a ranked, scoped plan, and next to it the observations — the Explain rail showing exactly why each case is there. She reads the reasoning, agrees with ten items, deletes one she knows is irrelevant, and adds one of her own. Eight minutes, not forty. The tax-rounding interaction that escaped two months ago? It's the second line of the plan, flagged as an uncovered edge case, because the Change Analysis Agent saw the diff touch the tax path and the Coverage Agent found no test guarding it.
The one action that stays gated
Here's the part I care about most, wearing my security hat. The plan generation is autonomous and read-only, and that's fine — reading a diff and a test catalog can't hurt anyone. But the Checkout squad also wanted the approved plan pushed back into their test-management tool and assigned to the on-duty tester automatically. That is a write, and writes get gated.
So when Priya clicks approve, the Mission doesn't quietly reach into the test-management system. It drops a row into the Decision Queue — "assign plan CT-8842 to tester queue, create 2 new edge-case stubs" — with the reasoning attached. The on-duty QA lead sees it, confirms, and only then does the write happen through the test-management MCP server. It's one extra click, and it's the click that means I can sign off on letting a Mission touch their tooling at all. Autonomous where it's safe, human-gated where it isn't. No exceptions, no "trust me."
The numbers, ninety days in
I don't ship a case study without measuring it, so here's what the Checkout squad saw over a quarter:
- Test-planning time per PR dropped from ~40 minutes to ~8 — QA now edits a plan instead of authoring one. Across ~40 PRs a week that's on the order of 20 engineer-hours reclaimed weekly, redirected into exploratory and adversarial testing that no automated plan can do.
- PR-to-merge cycle time fell by roughly a day on the median change, because the test-planning step stopped being a cold-start bottleneck sitting between "code approved" and "QA starts."
- Two production escapes avoided that we can directly attribute to the edge-case flagging — including a repeat of the exact tax-rounding interaction that had bitten them before. That class of bug is precisely what a rushed hand-built plan misses and a diff-grounded Mission catches.
- Zero unreviewed writes. Every action the Mission took against the test-management system went through the Decision Queue and a human approval. My audit log shows it.
Why it sticks
I've watched plenty of "AI for QA" tools get switched off after a month because they either over-promised autonomy and scared people, or under-delivered and became one more tab to ignore. This one stuck for a boring, structural reason: it puts the reasoning in front of the human instead of behind it. QA trusts the plan because they can watch it think on the Explain rail, and they trust the automation because the only actions with real blast radius wait for a human hand. The context that used to evaporate at the commit now survives the handoff — and it survives it observably, inside the team's own perimeter.
That's the pattern worth stealing whether or not it's test planning: find the place where your best people burn their sharpest hours reconstructing context a machine could have carried across, and let a StudioX Mission carry it — observably, and with humans on the actions that matter. It's the same AI workflow automation spine under every use case we ship; this is just the one I'd point a skeptical QA team at first.
Discussion
No comments yet — start the conversation.