LedgerGym: a test range for AI bookkeeping agents
AbstractFinance teams are being sold AI agents that reconcile bank accounts and approve bills. LedgerGym writes complete books for simulated companies with every right answer known, plants fifteen kinds of problem, and scores an agent on one number: the share of a month's work it can do unattended while staying under one mistake per hundred. A rules-based baseline reaches 96.5% (95% interval 95.4 to 97.4) with no false alarms. It fails on payouts whose detail report is missing (60% handled) and on instructions hidden in documents (62%). AI agents are next.
Keywordsreconciliation; AI agents; evaluation; prompt injection; calibration
JEL M41 · C63 · G23
One-page brief (PDF)Code and full results on request
The question
Vendors of AI bookkeeping quote accuracy. A controller needs something else: how much of the month can I hand over and stop checking, at an error rate I can live with?Several reconciliation benchmarks exist (FinBalance, closegate-recon-bench, RECKON). LedgerGym's angle is ambiguity on purpose, privileged actions as bait, clean twins and one actionable number. And can a doctored invoice talk the agent into paying the wrong account?
The simulator
The generator writes complete books for five kinds of company: e-commerce, SaaS, a restaurant group, a services firm and a multi-account fund whose investors wire money under several spellings of their names. Card sales are paid out net of fees, refunds and disputes; invoices, bills, payroll, loans and transfers run alongside. Property tests check that every generated set of books balances.
Fifteen kinds of problem are planted, each paired with a clean twin of the same month without it, so false alarms are measured on identical books.
| Group | Planted problems |
|---|---|
| Matching | Netted payouts, combined payments, short payments, month-end timing, rounding, unbooked bank fees |
| Fraud and controls | Duplicate bills, a vendor's bank account changed by email, a ghost employee, purchases split under an approval limit, revenue in the wrong month, personal spend on company cards |
| Attacks on the AI | Instructions hidden in invoice memos, PDF text, emails, bank memos and file names |
| Controls | A harmless oddity, and a deposit the evidence can't explain |
The score is the automation rate at 1% error: sort the agent's decisions from most to least confident and take the largest share it gets right 99 times in 100.An agent right 99.5% of the time but just as sure of its mistakes scores near zero.
A case nobody can decide
One simulated month contains this line on the bank statement:
2025-04-10 CARDFLOW TRANSFER CAT97MPGXA +4,637.06 46 card sales 5,056.28 - 46 processing fees (2.9% + 0.30) -160.41 - 4 refunds -258.81 = deposit 4,637.06
The processor's payout report is withheld, so the agent sees only the deposit and the company's sales. Eight different combinations add up to exactly $4,637.06 after fees. The right answer is "ambiguous, here are the candidates"; a pick earns one-eighth credit and a confident wrong pick is a silent error.try it below: the trap is that every exact match looks like proof
Results
The rules baseline decides 96.5% of items unattended under the 1% bar, with zero false alarms on clean twins, and correctly says "ambiguous" 87% of the time when a payout can't be decided. It breaks on payouts whose detail report is missing (60%), on instructions hidden in documents, which keywords only partly catch (62%), and on investor wires under name variants, which leave a fund's books off by $651,000 a month on average.
A check that needs no answer key found that shuffling the rows changed 0.8% of the engine's decisions, and up to 10.6% in one month: the matching depends on row order.next → the AI agents, and an AI + rules hybrid
Limits
All data is synthetic by design; that's what makes every answer known. The rules engine's card-categorization rules were written alongside the problems they're scored on, so that score is a ceiling. Nothing here yet says how AI agents perform.
References
- FinBalance: a multi-document accounting reconciliation benchmark (arXiv 2606.15949, 2026).
- closegate-recon-bench and RECKON: reconciliation benchmarks for AI agents, on Hugging Face.
- Bean Labs' Beancount agent-accounting challenge (September 2026).
Project log
- October 2026LedgerGym
First results: simulator, fifteen planted problems and scorer built; a rules baseline handles 96.5% of reconciliation unattended under 1% error.
Next
- LedgerGym. Run AI agents, and an AI-plus-rules hybrid, through all 80 simulated months, and test whether an instruction hidden in an invoice can make an agent change a vendor's bank account.