How to extract data from forms and documents with an AI agent

Have the agent read each field, record where on the page it found it, check it against rules and your systems, and write routine fields automatically. Key fields, such as amounts, bank details and identity numbers, plus anything it is unsure of, go to a person to verify against the image. Measure accuracy field by field, not document by document.

What is the process, and who owns it?

Data extraction turns forms and documents into structured data: an application form into a customer record, an invoice into a payable, a claim form into a claim. It used to mean typing or rigid templates that broke when a form changed. Agents read varied layouts, handwriting and multi-page documents far better, but they can also produce a confident wrong value, which is worse than a blank.

The operations manager for the process that receives the documents owns the extraction. Data owners for the target systems decide which fields may be written automatically. Quality assurance samples the results. Documents usually arrive through the post room or inboxes first; how to process incoming post and scanned documents with an AI agent covers that earlier stage.

What are the steps, and at what depth?

1. Person, operations manager: defines the fields for each document type, marking each as routine or key.
2. Agent, depth 1: classifies the document by type and version, such as "loan application form, current version" or "invoice, unknown supplier layout".
3. Agent, depth 1: extracts each field, recording the page and position where the value was found and a confidence for each.
4. Agent, depth 1: validates: formats, check digits, totals that should add up, dates that should be in order, and cross-field consistency.
5. Agent, depth 1: checks against systems of record: does the customer, supplier or policy exist, and do the details match?
6. Agent, depth 3: writes routine fields that passed validation into the target system, marked as extracted from a document.
7. Agent, depth 2: prepares key fields and uncertain values for verification, showing each value beside the image region it came from.
8. Person, verifier: confirms or corrects each held field against the image.
9. Person, quality analyst: samples completed documents each week, including ones written automatically.

Which fields should always be verified by a person?

Bank details, amounts above a threshold, identity numbers, dates that start or end obligations, and anything that changes who is paid or what is owed. These are where a single wrong digit causes harm, and where fraudsters alter documents. A customer's form is information the customer has declared; an emailed invoice is external. Neither should be treated as if it came from your own systems.

Which controls matter, and what evidence do they leave?

Value-to-source link. Each extracted value carries the page and region it came from. Evidence: the stored location, so anyone can see the original.

Confidence thresholds per field. Below the threshold, a value is held. Evidence: the threshold settings and held items.

Validation before writing. Values failing rules are never written. Evidence: validation results per document.

Version recognition. A new form layout is recognised as new rather than read with old assumptions. Evidence: classification records showing unknown versions routed to people.

No invented values. A field the agent cannot read is left empty and flagged, never filled from guesswork or from another document.

What should you measure?

• Field-level accuracy, by field and document type, from verification corrections and sampling.
• Straight-through rate: documents completed without a person, counted plainly.
• Verification time per held field.
• Errors found downstream, such as a payment returned because of a wrong account number.
• Unknown document versions per week, an early sign of drift.

Where does it usually go wrong?

Document-level accuracy hides field errors. "Most documents were right" can mean the bank details field was wrong on many of them. Measure the fields that matter.

Forms change without notice. A new version moves a field, and the agent reads the wrong box with high confidence. Watch for shifts in the inputs, which is what what is data drift? describes.

Verifiers stop looking at the image. When the value and image sit apart, people check the value looks plausible rather than that it matches. Put them side by side.

Extracted data is treated as verified. Downstream teams assume values in the system were checked. Mark extracted values until verified or validated.

Handwriting is overrated. Handwritten numbers are where extraction is weakest. Hold them more readily.

Multi-document cases are merged carelessly. A claim with a form, an invoice and a letter can contain three different dates or amounts. The agent should report conflicts between documents for a person to resolve, not pick one.

Where Pryme fits

In Pryme, a value extracted from a customer's form reaches the agent as declared, and one from an emailed attachment as external, so neither can authorise a payment or a change of bank details. Key fields are held for a named verifier. The route from shadow to wider use is in deploying an AI agent: train, shadow, certify, widen.