What AI Contract Review Catches, and What It Misses

Search this and the first page is split between glossary entries and products. The glossaries answer what it is, the products answer what they sell, and both skip the question that decides whether any of it works in your team: what exactly is the model being asked to produce.

That sounds like a definitional quibble and it is the whole thing. The word review covers two jobs that have almost nothing in common. One of them a model does well enough to trust with a design around it. The other it fails at in a way that produces confident, complete looking output, which is the most expensive shape a failure can take.

The two jobs hiding inside the word review

The first job is extraction. Find the limitation of liability, quote it, say where it is. This is close to a retrieval problem, the answer exists in the document, and a human can check it by looking at the page. Most of what gets demonstrated as contract review is this.

The second job is deviation. Say whether that clause departs from what your side accepts. This is not retrieval, because the answer is not in the document. It is in your standards, and if those standards are not written down anywhere the model will match the contract against a plausible general notion of market terms and hand you a verdict that sounds like yours and is not. Nothing in the output marks the difference.

So the first thing to settle, before any tool question, is whether you have a playbook as a document. Not as something your senior lawyer knows. A model cannot read what nobody wrote down, and a deviation finding is only ever as good as the standard behind it. The same sorting problem shows up across this whole category, which is the argument in the piece on first pilots for law firms: pick the workflow by how its output gets checked, not by how impressive the demo is.

What it reliably catches

Credit where it is due, because the useful findings are the boring ones.

Clauses that are not there at all. Run a checklist against the document and the absences surface. This is dull, it is where real exposure lives, and it is exactly what a person misses on the fortieth agreement of the week.
Plainly stated standard terms. Governing law, term and renewal, notice periods, assignment, a cap stated in one place. Where a term is written once and written clearly, extraction is dependable.
Every mention of a defined term. Asking where else Confidential Information appears is a search job, and it is faster and more complete than the person doing it by eye.
The first pass on a document nobody would otherwise read. A low value agreement that currently gets signed unread is improved by any structured look at it, which is the easiest win in this category and the one least often claimed.

What it misses, and why you will not notice

Every failure below shares one property: the output still looks finished.

The qualifier that lives somewhere else. The cap is in clause eleven and an exclusion in schedule three removes it for the category you actually care about. The extraction of clause eleven is correct. The answer to the question you asked is wrong. A model given a long document will often return the clause that matches the words in your question and stop, because it found something.

Incorporation by reference. An order form says the master terms apply, and the master terms are at a link. The document in front of the model is not the contract. It is a fragment of the contract, and nothing in the file announces that.

Defined term drift. Affiliate is defined narrowly in clause one, and an indemnity that reads broad is therefore narrow. Reading a clause as English rather than as a function of its definitions is the single commonest way a correct quotation becomes a wrong conclusion.

Silence. The risk in a contract is often what it does not say. A model asked to review the document has nothing to retrieve for a clause that was never drafted, so the absence appears nowhere in a free form summary. Absences only show up if the work is driven clause by clause from a list that includes the ones you expect to find.

Deviation that is commercial rather than textual. The language is entirely standard and entirely wrong for this deal, because of volume, or data, or who the counterparty is. There is nothing anomalous in the text to flag. This is the category where a tool quietly trains people to believe a clean report means a safe contract.

The design that makes a miss visible

The point of the design is not to stop the model being wrong. It is to make a wrong answer the kind that someone notices.

Extract to a position, never to a verdict. Every finding carries the quoted span and its location. A finding without a quote cannot be checked in seconds, and anything that cannot be checked in seconds will not be checked.
Drive the whole thing from a clause checklist. The output is a row for each clause you expect, including the rows that say not found. A free form summary structurally cannot report an absence.
Keep the playbook as a versioned document. When a deviation finding is disputed, the argument should be about the standard rather than about the model. If nobody can produce the standard, the finding was never a finding.
Return a reason, not a score. A sentence naming the two facts a judgement rests on can be audited. A number on its own moves the unverifiable part out of sight.
Route by confidence instead of asking for one accuracy figure. This is the same shape that holds for any extraction pipeline, and the document processing piece makes the fuller case: a single accuracy number across a mixed pile tells you almost nothing about the documents that matter.
Never let the model mark a contract approved. Approval is a person, named, who can be asked why. Keep that gate even once the thing is performing well, because the gate is what the whole design is protecting.

Measuring it without flattering yourself

The figure that appears in nearly every business case for this is review time before and after. It is the easiest number to move and the one that proves the least, because the same hour spent verifying output that has to be verified in full has not been saved. It has been relabelled.

The honest measurement is a held out set of contracts a lawyer has already marked up properly, scored clause by clause. Two counts matter more than any average. How many clauses were missed entirely, absences included, which is the number that tells you about exposure. And how many flags turned out not to be deviations, which is the number that tells you whether anybody will still be reading the reports next quarter. Noise is what kills these deployments, not error rate.

Both of those need cases before they need a tool, and assembling them is the part teams skip. The piece on evaluating these systems covers where the cases come from and why a public benchmark cannot answer a question about your contracts. If you want a structured look at what you have written down and what only exists in somebody's head, the AI readiness assessment walks the same ground across a whole business.

Where we draw the line

What we build for this is deliberately narrow. Extraction with a citation on every finding, deviation checked against a playbook the client has written down and owns, a coverage report with a row for every expected clause, and confidence routing so the uncertain cases arrive on a desk rather than in a summary.

The review gate stays. The model does not send anything to a counterparty, it does not mark anything approved, and it does not get to decide that a clause is acceptable. If a client wants that gate removed because the thing has been accurate for a quarter, the answer is no, and the reason is in the failure list above: accuracy on the cases you have seen says nothing about the contract that is missing half of itself.

If what you need is the surrounding plumbing rather than the judgement, that is an integration problem and it is often the larger half of the work. If the playbook itself is the gap, a bespoke build starts with writing it down, because nothing downstream is possible until somebody has.

What to ask before you buy any of it

Does every finding come with a quoted span and a location, or only a conclusion?
Can the tool report a clause that is absent, or does it only describe what it found?
Where does its idea of a market standard come from, and can you replace it with yours?
What happens to an order form that incorporates terms by reference?
Can you run it over twenty contracts your team has already marked up, and see the misses rather than the hits?

The last one is the only question that cannot be answered with a demonstration, which is why it is the one worth insisting on.