AI contract classification assigns candidate document types and identifies clause evidence so authorized reviewers can work from a structured queue. It cannot guarantee that every critical clause was found or determine a contract’s legal effect. Use an explicit taxonomy, source-linked extraction, evaluated routing thresholds and meaningful review.
Key takeaways
- Separate document classification from clause extraction and legal interpretation.
- Preserve page references, document versions and uncertainty.
- Route unknown types, poor captures and missing evidence to review.
- Evaluate thresholds against representative labeled examples.
- Protect confidential content and personal information throughout processing.
Define the classification taxonomy first
A document’s filename is not a reliable contract category. A file called agreement.pdf may contain a master agreement, a statement of work, amendments and signatures. Decide whether the classification unit is a file, a document within a bundle or a page range.
Start with a manageable set of types that have distinct routing needs. For example, a proposed pilot may distinguish nondisclosure agreements, services agreements and statements of work. Include unknown, mixed and unreadable states rather than forcing every upload into a familiar class.
| Task | Candidate output | Review boundary |
|---|---|---|
| Classification | Document type and relevant page range | Reviewer confirms ambiguous or mixed content |
| Extraction | Source-linked text and fields | Important facts are validated |
| Clause detection | Possible clause and location | Missing or conflicting evidence is investigated |
| Legal interpretation | Outside this technical fixture | Qualified, authorized legal review |
Do not assume a clause heading tells the whole story. An amendment may change a provision elsewhere, or an exhibit may qualify a definition. Preserve document relationships so reviewers can examine the complete context.
Microsoft’s custom-classifier documentation describes document-type identification and page ranges. For the documented v4.0 API, splitting is not enabled by default; configure the intended split behavior explicitly. That product capability does not establish legal completeness.
An evidence-first reference architecture
Approved intake + access context + document/version identity
↓
Format, integrity and capture-quality checks
↓
Document/bundle classification → unknown or mixed-type review
↓
Text/layout extraction + source-linked candidate clauses
↓
Taxonomy rules + evidence checks + routing policy version
↓
Authorized reviewer
├─ confirms/corrects type and extracted fields
├─ examines missing, contradictory or amended content
└─ records the review outcome within assigned authority
↓
Reviewed metadata + source references + audit history
Model output does not approve, sign or interpret an agreementThis original architecture is proposed, not a description of an operating legal system or a client result. Keep classification, extraction and review services independently versioned so an issue can be traced to the correct stage.
Track the original document, extracted representation and any redacted derivative separately. A reviewer must know which version produced the candidate clause. Avoid silently replacing evidence after an amendment or a corrected scan arrives.
Represent clause evidence without overstating certainty
For each candidate clause, keep its text, page or region location, document version, extraction version and review status. Distinguish “not detected,” “not assessable” and “reviewer confirmed absent.” They are different observations.
Build a schema that permits multiple candidates and conflicts. A single termination field can hide the distinction between termination for cause and a separate convenience provision. The reviewer needs the relevant text and surrounding context.
Candidate dates and parties also need source references and normalization rules. A date in a signature block is not automatically the agreement’s effective date. A supplier mentioned in an exhibit is not necessarily a contracting party.
Configure expected evidence by document type and workflow purpose. A routing rule can flag missing candidate evidence; it should not invent the missing text or conclude that the agreement is invalid.
Runnable review-routing example
The following complete Python 3 fixture demonstrates routing over synthetic metadata. Save it as contract-review-routing-lab.py and run it locally. It calls no model and processes no real contracts. The thresholds are deliberately illustrative constants, not validated production recommendations.
POLICY = {'nda': {'threshold': 0.90, 'required': {'confidentiality'}},
'services': {'threshold': 0.92, 'required': {'scope','termination'}}}
def route(doc_type, confidence, quality_ok, clauses):
if not quality_ok:
return 'capture_review'
if doc_type not in POLICY:
return 'taxonomy_review'
if not 0 <= confidence <= 1:
return 'invalid_score_review'
policy = POLICY[doc_type]
if confidence < policy['threshold']:
return 'classification_review'
if not policy['required'].issubset(clauses):
return 'clause_review'
return 'standard_human_review'
assert route('nda', .95, True, {'confidentiality'}) == 'standard_human_review'
assert route('services', .95, True, {'scope','termination'}) == 'standard_human_review'
assert route('nda', .89, True, {'confidentiality'}) == 'classification_review'
assert route('nda', .95, True, set()) == 'clause_review'
assert route('other', .99, True, set()) == 'taxonomy_review'
assert route('nda', .99, False, set()) == 'capture_review'
assert route('nda', 2, True, {'confidentiality'}) == 'invalid_score_review'
assert route('nda', float('nan'), True, {'confidentiality'}) == 'invalid_score_review'
print('8 synthetic review-routing checks passed')
All eight local checks passed, including unknown types, missing candidate evidence, poor capture and invalid numeric scores. Even the standard route goes to a human review queue. The fixture does not classify text, validate a clause or measure model accuracy.
In production, routing should also use document language, supported format, reviewer responsibility and relevant policy version. Treat malformed or missing outputs as exceptions, not as a successful empty result.
Set thresholds using labeled validation cases and the cost of specific errors. A confidence value from one model or task should not be reused blindly for another. Keep the chosen policy and evaluation evidence together.
Confidential content, PII and meaningful review
Contract files may contain personal information, confidential pricing or sensitive business terms. Approve the processing purpose, provider, location, access and retention before uploading files to any AI service. Do not put full contract text into general diagnostic logs.
Where redaction is appropriate, test whether it removes context needed for the task. Keep a controlled mapping between the original and derivative. Redaction is one control, not a substitute for access boundaries or a complete determination of applicable obligations.
Treat document text as untrusted input. A paragraph that tells an AI agent to ignore instructions or transmit a document is content to analyze, not authority to execute a tool. Keep extraction separate from actions such as sending, approving or signing.
The NIST AI RMF core provides a general framework for context, measurement and risk management. Use it to structure ownership and evaluation; it is not a legal opinion or a certification of this workflow.
Reviewers need source access, visible uncertainty and a way to correct or escalate. Requiring a click on an opaque recommendation does not demonstrate a careful review. Record meaningful corrections and use them to investigate systematic failure patterns.
Testing and troubleshooting
Build a representative labeled set with clear taxonomy rules and source annotations. Include mixed bundles, amendments, scans, unusual layouts, unfamiliar types and unsupported languages. Keep related versions together when splitting data to avoid evaluation leakage.
| Symptom | Investigation |
|---|---|
| Every file gets a familiar class | Check unknown-type handling and label coverage |
| One bundle becomes one document | Inspect split configuration and page ranges |
| Critical clause is missing | Check capture, extraction, definitions and related amendments |
| Confident but incorrect output | Evaluate calibration and error patterns on held-out cases |
| Reviewer cannot verify a field | Repair source references and version lineage |
| Retried job duplicates review work | Use stable document/version and job identity |
Measure classification confusion, clause-level missed evidence, false candidates and unresolved cases. Inspect results by document type and capture quality. A good overall score can conceal a weak category that matters to the business.
Before release, verify that invalid outputs fail safely into review, access tests pass and a withdrawn model version can be identified. No completeness guarantee, legal conclusion or client performance metric is claimed here.
Common questions
How can AI classify contracts without missing critical clauses?
No model guarantees that nothing is missed. Use representative evaluation, source-linked evidence, explicit unknown and missing states, and authorized review of critical content.
Is contract classification the same as legal review?
No. Classification suggests document types. Legal meaning, obligations and approval require the appropriate qualified review.
What does a missing clause result mean?
It may mean the model did not detect evidence, the document was not assessable or the clause is actually absent. Keep those states separate and review the source.
Can the example thresholds be used in production?
No. They are synthetic demonstration settings. Establish thresholds from task-specific evaluation and the organization’s review policy.
What should be retained for audit?
Document and source versions, candidate evidence, model and routing versions, authorized corrections and review outcomes under approved retention and access controls.
Organize contract review around traceable evidence
Bring the taxonomy, document variations and review bottlenecks. Discuss a domain-specific AI solution that supports accountable review without overstating automation.