Replay it against periods you have already closed. Give the agent read access only, feed it the source documents from a past month, and compare every entry it proposes with what your team actually posted and the auditors accepted. Score each answer, study the misses, and grant posting rights one capability at a time once the scores hold.
Why is the general ledger a special case?
Errors in a support chat stay in a conversation. Errors in the general ledger flow into the trial balance, the management pack, tax returns and the statutory accounts. A wrong cost centre is cheap to fix today and expensive after the board has read the numbers.
The ledger also has a memory problem. Reversing an entry leaves both the original and the reversal on the record, which is correct, but a run of agent mistakes clutters the audit trail and erodes the controller's trust. Testing before the first write is cheaper than cleaning up after it.
What should the test cover?
Test each capability separately, because each needs a different depth.
| Capability | What you replay | What counts as a pass |
|---|---|---|
| Invoice coding | A month of supplier invoices | Account and cost centre match what was posted |
| Accrual drafting | Month-end accrual workpapers | Amount within tolerance, support attached |
| Bank reconciliation | Bank statements and ledger | Every unmatched item explained |
| Intercompany | Both sides of each balance | Mismatches found, not forced |
| Journal narrative | Posted journals | Narrative traces to a source document |
Include the hard cases on purpose: credit notes, partial deliveries, a supplier that changed its name, a foreign-currency invoice. A test set made only of clean invoices proves very little.
What are the steps, in order?
- Grant read access only. The agent reads the ledger and the sub-ledgers that feed it, but cannot write. Our guide on giving an AI agent read access to your general ledger covers scoping.
- Pick two or three closed periods. Choose months with the usual mix, and one with a known messy event such as a system migration.
- Replay. The agent proposes entries from the source documents. Nothing is posted.
- Score against the posted record. Compare each proposal with what was posted. Where the posted entry was itself wrong, note it; your history is the benchmark, not gospel.
- Test the refusals. Give it items it should not touch, such as a reopening request or a payment to a changed bank account, and check that it declines and says why.
- Review the misses with the controller. A pattern of misses on one account usually means a missing source, not a bad model.
- Run in shadow. For a live month, the agent proposes alongside the team. Compare daily.
- Set the agent's depth from the evidence. Coding may run at depth 3 while accruals stay at depth 2.
A worked example
Take a services firm with 2,000 supplier invoices a month. The agent replays January and February. Coding matches the posted account on nearly all routine invoices, but it repeatedly codes software subscriptions to IT when the firm charges them to the department that uses them. The controller adds the recharge policy as a source, and the March replay clears.
Accruals tell a different story. The agent's amounts are close, but it attaches no support for two estimates. Accruals stay at draft until it does.
What stays with a person, even after testing?
The controller decides what a pass means and signs the result. Posting above materiality, period reopening, intercompany eliminations and anything involving judgement about revenue recognition stay with people regardless of scores. Testing earns the agent routine work; it does not earn it judgement.
Which controls make the test trustworthy?
- Your cases, not a vendor's. A demo on generic invoices tells you nothing about your chart of accounts. The same principle applies to testing an AI support agent before launch: the test set must come from your own operation.
- A written floor before you start. Agree the pass mark per capability in advance, so nobody moves it after seeing results.
- A filed record. Every proposal, score and refusal is kept, so an auditor can see what the agent was tested on.
- Retesting. A new entity, a new chart of accounts or a model change means a new replay.
The topic guide AI agents for finance: books you can audit shows how testing feeds the wider control design.
Where Pryme fits
Attest does this replay as certification. It runs the agent against real cases from your own ledger and scores each answer against a floor across seven gates. The agent earns one depth, the certificate lapses after 90 days, and a run that clears no depth drops the agent to read-only the same day.
Questions people ask
How many periods should we replay?
Enough to cover your normal mix and at least one unusual event. Two or three months usually show the patterns; a year-end month is worth including.
Can we test in a sandbox ledger instead?
A sandbox is useful for testing writes, but its data rarely reflects your real history. Replay on real closed periods with read access, then use the sandbox to check the posting mechanics.
Who should sign off the test?
The controller or the process owner for the ledger, not the team that built the agent. The person who answers for the numbers decides whether the agent may touch them.