How to run an AI agent readiness assessment in a law firm

Choose one practice group and assess paralegals, trainees and associates who will check an agent's work, plus supervising lawyers who sign it off. The test, set in legal work, probes provenance, privilege, held actions and the record. The report names the gaps, decides whether the first agent reviews documents or drafts intake notes, and sets what to train first.

What is the assessment for in a law firm?

Law firms tend to start with agents that read: summarising disclosure, building chronologies, extracting clauses from contracts, and drafting matter intake notes from a client's first email. Every output is reviewed by someone who answers, ultimately, to a supervising lawyer bound by professional duties of competence, confidentiality and supervision.

The assessment tests whether those reviewers understand an agent well enough to supervise it the way they would supervise a trainee. The head of legal operations usually owns it, with a sponsoring partner from the practice group. For the concept, see what a readiness assessment is.

Who should take it?

A single practice group, up to 25 people. Litigation is a common first choice, because disclosure review is document-heavy and well defined. Real estate and corporate are others.

The cohort should include paralegals, trainees, associates up to the level that supervises them, and the knowledge lawyer or practice support lawyer who maintains precedents. Partners sit separately. Their role is to sign and supervise, and they answer differently when juniors are present. The order of training is discussed in who in a law firm needs AI agent training first.

The steps

1. Person (head of legal operations): chooses the practice group and lists what the agent will never do: advise a client, file anything with a court, or communicate with the other side.
2. Person (head of legal operations): asks for the test set in legal work, with questions on documents, privilege and supervision.
3. Person (each member): sits the test alone.
4. Person (head of legal operations and sponsoring partner): read the report by concept and seniority.
5. Person (sponsoring partner): chooses the first agent and its depth, usually reading at depth 1 and drafting at depth 2.
6. Person (knowledge lawyer): trains the weakest concept using the group's own document types.
7. Agent, depth 1: reviews a closed matter's documents in shadow, tagging relevance and possible privilege for comparison with the review the firm actually did.
8. Person (cohort): reassesses before the agent works on a live matter.

What does the test usually find in law firms?

Provenance of documents. Lawyers are trained to question documents. But in the test, many treat a document produced by the other side as equal to the client's own records. In governed terms, the client's document management system is sealed or signed, the client's account is declared, and anything from the other side is external. The agent labels each; reviewers must read the labels, particularly when a chronology mixes sources.

Refusals and withheld actions. Junior staff sometimes expect the agent to send a draft letter if asked firmly. It will refuse anything outside its bounds, and the firm's own limits will keep sending with a lawyer. Understanding that a refusal is correct behaviour avoids wasted escalations.

The record and privilege. Every verdict is filed. Reviewers should understand what the record contains, who can see it, and how the firm's own confidentiality and privilege controls apply to it. This question usually exposes the widest spread of answers.

What does the report change?

It decides the first agent. A group weak on provenance should start with an agent that drafts intake notes from the client's own email, where sources are few, rather than one building chronologies from mixed disclosure.

It changes supervision. If associates score below paralegals on reading held actions, associates must train before they supervise agent output, or the firm's supervision duty is met on paper only.

It shapes the knowledge function. If the test shows confusion about which precedents the agent uses, the knowledge lawyer takes ownership of the agent's sources.

What evidence should the firm keep?

The never-do list, approved by the sponsoring partner.

The report by concept and seniority.

The decision on the first agent, with reasons.

The reassessment before live matters.

What should the head of legal operations measure?

• Scores by concept, by seniority.
• During shadow, agreement between the agent's tags and the firm's review, especially on privilege calls.
• After go-live, how often supervising lawyers change agent-assisted work, and why.

Where does it go wrong in law firms?

Assessing the innovation committee instead of the practice group. The people keenest on agents are not the ones who will review its work at 22:00 before a disclosure deadline.

Partners exempting themselves. The supervising lawyer's understanding sets the ceiling for the whole group.

Starting with privilege review. Privilege calls are high-stakes and judgement-heavy. They should come after the team has shown it reads provenance and holds correctly on lower-risk work.

Accountancy and property firms run similar professional-services assessments; compare the accountancy firm version and the property management firm version. Partners' own needs are covered in what AI agent training a law firm board needs, and the topic guide is training teams for AI agents.