How to run calibration sessions for paralegals

Paralegals calibrate on the document calls an agent proposed and held: relevance, possible privilege, redactions and key dates. Give each paralegal the same small set of documents from a closed matter, with the agent's tags hidden, and have them decide alone. Reveal the agent's tags, compare, and let the supervising lawyer set the gold standard for the review protocol.

Why do paralegals need calibration?

In disclosure and document review, an agent typically reads each document, proposes tags for relevance and issues, flags possible privilege, suggests redactions of personal data, and extracts dates for a chronology. It works at depth 1 and 2: it reads and proposes; paralegals confirm or change every tag on held documents, and a lawyer signs off the review.

Review teams are notoriously inconsistent. One paralegal tags a forwarded email chain as privileged because a lawyer appears in it; another tags only the parts seeking legal advice. One redacts every personal phone number; another only those of non-parties. Inconsistent review creates risk in litigation and teaches the agent nothing reliable. The general method is in how to run calibration sessions for AI agent reviewers.

The supervising associate or partner on the matter owns the standard. A senior paralegal facilitates. A knowledge lawyer helps turn standards into the review protocol.

Which documents belong in the set?

Twelve to fifteen documents from a closed matter, chosen with the supervising lawyer:

• An email chain where a lawyer is copied but no legal advice is sought.
• A board pack with one paragraph summarising legal advice.
• A document relevant to one issue but marked confidential by a third party.
• A spreadsheet with personal data of employees not involved in the dispute.
• A letter with an ambiguous date that matters for a limitation point.
• A near-duplicate with handwritten notes on it.
• A document the agent tagged irrelevant that a careful reader would include.

The session, step by step

1. Person (senior paralegal): selects documents with the supervising lawyer's approval and checks the matter is closed.
2. Person (each paralegal): tags each document alone without seeing the agent's tags: relevance, issues, privilege, redactions, key dates.
3. Agent, depth 1: reveals its proposed tags and the passages it relied on, with the provenance of each document: client's own systems, the other side's disclosure, or third parties.
4. Person (facilitator): reveals the spread on each tag.
5. Person (paralegals): explain their reasoning, quoting the passage that decided it.
6. Person (supervising lawyer): sets the gold standard.
7. Person (knowledge lawyer): writes it into the review protocol and sends it to the brief owner.
8. Person (senior paralegal): samples reviewed documents on the next live matter.

Why hide the agent's tags first?

If paralegals see the agent's tag before deciding, most will agree with it. The session then measures agreement with the agent, not the team's own judgement. Hiding the tags shows where people differ from each other and from the agent, which is what the supervising lawyer needs to see.

What does a gold standard look like for review?

Specific to the matter type, written for the protocol. For example: "A lawyer appearing in an email chain does not make it privileged. Tag as privileged only the messages seeking or giving legal advice; tag the rest of the chain normally and flag for lawyer review."

On redaction: "Redact personal contact details of individuals who are not parties or witnesses. Do not redact names."

On dates: "Where a date is ambiguous, record both possible dates and flag the document for the chronology lawyer. The agent proposes both and does not choose."

What does a session look like?

Take a hypothetical litigation team of six paralegals. On the board pack, two tagged the whole pack privileged, three tagged nothing, and one tagged the single paragraph. The supervising associate explained that only the paragraph summarising advice was privileged and should be redacted, with the rest produced. The protocol was updated and the brief owner changed the agent to flag paragraphs, not whole documents, where advice is summarised.

Which controls does calibration support, and what evidence does it leave?

Consistent review. Evidence: spreads per tag over sessions.

Privilege protection. Evidence: the privilege standard in the protocol.

Data protection. Evidence: the redaction standard and sampled documents.

Agent alignment. Evidence: protocol and brief changes.

What should the supervising lawyer measure?

• Agreement on privilege and relevance tags.
• Documents re-tagged at second-level review.
• Agent tags overturned by paralegals, by tag type.
• Redaction errors found before production.

Where does it go wrong with paralegals?

Using live matters. Never calibrate on documents still under review in an active case.

No lawyer setting the standard. Paralegals agree among themselves on a privilege rule the lawyer would not accept.

One protocol for every matter type. Standards differ between employment, commercial and regulatory matters. Calibrate per type.

Ignoring the documents the agent marked irrelevant. Reviewers rarely see them, so errors there go unnoticed. Put at least one in every set, and sample the irrelevant pile on live matters as well as the relevant one.

Customer support and collections teams calibrate on different material with the same method; see calibration for customer support agents and for collections agents. For planning, see the AI agent training plan template for paralegals and how long it takes to train paralegals to supervise AI agents. The topic guide is training teams for AI agents.