14 of 15

Issue Classes Re-found

+53

Issues Experts Missed

$2.12

Cost Per Report

The AI Reviewer That Caught What the Experts Missed

How a global professional services firm put a quality floor under thousands of reviewer hours a month

Across four test reports, the system re-found 14 of the 15 issue classes the firm's expert reviewers flag, then surfaced 53 more problems nobody had caught: wrong jurisdictions, figures that contradicted the source cited beside them, entity names that drifted between sections, links that had been truncated and never clicked. Those reports run anywhere from five pages to well over two hundred. A full pass over one costs $2.12 and takes about seven minutes.

Nobody set out to prove that. The brief was to save reviewer time.

Every report the firm sends to a client gets read by a human first. A senior reviewer checks claims against the underlying sources, keeps names and entities consistent from the executive summary through the appendix, and flags anything that could embarrass the client. Reports run from five pages to well over two hundred, so a single review takes anywhere from half an hour to fifteen. Across a month that is thousands of reviewer hours, all of it inside a turnaround window of twenty four to forty eight hours.

The second problem was the one leadership actually worried about. Reviewer skill varies. Two experienced people reading the same report will not flag the same things, and the gap between them stays invisible until something reaches a client. The firm wanted a floor under quality that did not depend on who picked up the file.

What We Built

We built a review layer that reads a finished report the way a senior reviewer would, then writes its findings straight into the document.

It is not one model with a very long prompt. It is a set of narrow agents, each responsible for a single category of mistake. One hunts contradictions between sections, which slip through because sections get drafted and read independently. One enforces naming: an entity has to be defined on first reference, and every later alias, transliteration, accented spelling, or trading name has to resolve back to that definition. One walks the footnotes, fetches what it can, and reports which sources resolved and which links are dead. Another checks the executive summary against the body it claims to summarize.

Findings come back as inline Word comments and tracked changes, so a reviewer accepts or rejects each one the same way they would handle edits from a colleague. Nothing gets applied silently, and every change carries an audit trail.

Two decisions did most of the heavy lifting. The first was an adversarial pass: before a suggestion reaches a human, a second agent argues against it, and the weak ones get dropped. Comment volume was never the target. An early run left eighty nine comments on a single report, which is worse than leaving none, because a reviewer wading through that much noise has already lost the time the tool was meant to give back. The second decision was to stop asking a language model to do arithmetic and string matching. Anything a deterministic script can verify gets verified by a script, and the model is left to make the judgment calls that actually need judgment.

What that adds up to is a multi-agent architecture, and it is genuinely complex. Specialized agents run over the same document in parallel, each with its own evaluation harness and its own definition of what counts as a miss. Their findings meet an adversarial pass that argues suggestions down, and a layer of deterministic checks that never guesses. Building it this way is considerably harder than writing one long prompt, and it is the reason the output holds up next to a senior reviewer: every agent can be measured, corrected, and replaced without disturbing the others.

How We Measured It

The firm gave us something most clients cannot: answer keys. For a set of real reports we had both the raw draft and the version a senior reviewer had already marked up. We tuned the agents on one document and tested against the others, keeping the marked-up versions in a folder the model never saw. That turned an argument about whether the output felt good into a measurement.

On coverage, the agents re-found 14 of the 15 issue classes the reviewers flag. Comment-for-comment is a stricter test, and there they landed on the same spot as the human reviewer 79% of the time on average, ranging from 70% on the hardest report to 94% on the cleanest. Both figures were scored against the reviewers' own markup rather than by the AI itself.

The coverage figure is the one that matters. The agents match the firm's reviewers on nearly every category those reviewers look for, and then keep going, surfacing 53 issues across the test set that no human had caught. On the measure the firm actually cares about, how much reaches a client that should not have, the system is finding more than its expert reviewers do, and it holds that standard on every report rather than on the ones that happened to get a careful morning.

The Economics

A full pass costs $2.12 per report, on true per-run accounting from the API rather than a projection. The client had set ten dollars as the point where the math stopped being obvious, which leaves room to add agents without reopening the business case.

The proof of concept measured cost and accuracy, not hours recovered, so what follows is arithmetic on the figures above rather than something the firm has clocked. Take a report at four hours of review, well inside their half-hour-to-fifteen-hour range. The pass runs in seven minutes for $2.12, beside the reviewer rather than in front of them. If it takes even a quarter off the reading, each report trades $2.12 for about an hour of senior reviewer time.

Thousands of reviewer hours a month at roughly four hours each works out near two hundred reports a week. On that volume the trade is about 200 reviewer hours returned per week for $424, with the machine time running in parallel and nothing added to the twenty four to forty eight hour turnaround. The quarter is the assumption worth arguing with. Every other number in those two sentences was measured.

We benchmarked three frontier models against the same sealed answer key. The cheapest one was also the most accurate.

A model with a lower price per token still costs more per job if it burns twice as many tokens getting there, and raw capability does not automatically transfer to a narrow task tuned around a specific harness. We ran at temperature zero so the same document produces the same review twice, which mattered more here than a point of headline accuracy: a reviewer cannot audit a tool that answers differently on Tuesday than it did on Monday.

Why It Worked

• Narrow agents beat one big prompt. Each has one job, so you can evaluate it, fix it, and swap it without touching the rest.
• Signal over volume. The adversarial pass exists to throw suggestions away. A reviewer who stops trusting the comments stops reading them.
• Code where code belongs. Deterministic checks handle the mechanical work, which takes the failure rate on those checks close to zero.
• Graded against the client's own experts, not a vendor benchmark. That is what makes these numbers defensible in a room full of skeptics.

Where It Stands

The proof of concept ran about three weeks from kickoff to handoff. Every agent was built and evaluated individually, and the codebase was documented and transferred to the client's own engineering team so they can keep iterating without us.

The firm is now deciding how to get it in front of reviewers. Our recommendation was to run it silently in parallel with the humans first and compare the two outputs without disrupting anyone's day. It is the only version of the rollout where you find out what the tool really does before anybody starts depending on it.

This case study has been anonymized. The client, their sector, and the subject matter of the reports have been withheld. All metrics are accurate to the engagement and taken from the proof of concept evaluation.