{"markdown":"# Best AI tools for forensic accountants: testing LLM extraction against printed balances\n\nA language model can misread one digit on a bank statement and still hand back perfectly valid JSON. Every field is a string where a string is expected, every amount is a number, and jsonschema.validate() comes back clean. The file is wrong and the validation says it is fine.\n\nThat gap is the whole problem with choosing AI tools for forensic work on the strength of a demo. Structure is cheap to produce and easy to fake. Agreement with the printed page is neither. Forensic accounting already has a control for this and it predates LLMs by a century: the statement proves itself. Opening balance plus credits minus debits equals closing balance. The transaction column sums to the printed total. The row count matches the printed count. If an extraction satisfies those constraints using figures the model did not generate, you have evidence. If it doesn't, you have a rejected file.\n\nThis guide is for teams reconciling statements at volume — cross-border finance ops, SaaS billing teams, and store operators matching processor payouts to bank deposits. It is a test you can build in an afternoon and point at any vendor or model.\n\n

Figure: Editorial context image from the source page (preview only; rights remain with the publisher) · Credit / further reading: dev.to
\n\n## Why schema validation is not a test\n\nA schema checks shape, not truth. It will happily certify {\"closing_balance\": 48219.07} when the statement prints 42819.07, because both are decimals. It will also certify a row set that is internally consistent but transposed, a currency code that is valid ISO but wrong for the account, or a date that parses cleanly but falls outside the statement period.\n\nThe failure mode is not a crash. It is a plausible, well-formed, quietly wrong ledger that flows downstream into a working paper, a VAT reclaim, or a board pack. Every one of those downstream systems trusts the JSON because a schema said it was fine.\n\nThe printed balances are the only ground truth in the room that the model did not author. Use them.\n\n## Step 1: Harvest the controls before you prompt\n\nFor each statement in your test set, record by hand — or with a human-reviewed OCR pass — the figures the page actually prints:\n\n- Opening balance and closing balance, with sign convention noted (credit-positive vs debit-positive)\n- Printed total debits and total credits, if shown\n- Printed row count\n- Statement period start and end\n- Account currency, and the currency of each row if the statement is multi-currency\n- Any FX rate lines and their stated base/quote\n\nThis is the boring step and it is the one that determines whether your whole harness is worth anything. Twenty-five statements is enough to start. Fifty is better.\n\n## Step 2: Freeze a golden set that includes the ugly ones\n\nPull statements that break things, not statements that photograph well. For cross-border and store teams specifically, force these into the set:\n\n- Comma-decimal locales (1.234,56) next to dot-decimal ones in the same batch\n- Statements spanning a page break mid-table\n- Multi-currency accounts with FX conversion lines that are not transactions\n- Processor payout summaries (Stripe, Shopify Payments, Amazon) landing as single net deposits against dozens of gross sales\n- Scanned or skewed PDFs with no text layer\n- Two rows that look like duplicates but are not (same amount, same day, different reference)\n\nStore the source file hash alongside the expected values. When a vendor updates their model, you want to know exactly which document changed behaviour.\n\n## Step 3: Write the harness\n\nThree stages: prompt, parse, assert. The assertion stage is the part vendors tend to skip in their own marketing. Roughly:\n\npython\nfrom decimal import Decimal, ROUND_HALF_UP\n\nq = lambda x: Decimal(str(x)).quantize(Decimal('0.01'), ROUND_HALF_UP)\n\ndef check(stmt, rows):\n credits = sum(q(r['amount']) for r in rows if r['direction'] == 'credit')\n debits = sum(q(r['amount']) for r in rows if r['direction'] == 'debit')\n results = {\n 'balance_chain': q(stmt['opening']) + credits - debits == q(stmt['closing']),\n 'credit_total': q(stmt['printed_credits']) == credits,\n 'debit_total': q(stmt['printed_debits']) == debits,\n 'row_count': stmt['printed_count'] == len(rows),\n 'period_ok': all(stmt['start'] <= r['date'] <= stmt['end'] for r in rows),\n 'no_dupes': len({r['reference'] for r in rows}) == len(rows),\n }\n return results # every value must be True to auto-clear\n\n\nRun every statement through every candidate tool and record the boolean vector, not just a pass/fail. A tool that fails row_count but holds balance_chain is telling you something different from one that fails balance_chain outright.\n\n

Figure: HEXOPS original illustration inspired by the topic (not a copy of the source article imagery) · Credit / further reading: dev.to
\n\n## Step 4: Assert arithmetic, not structure\n\nNote what the harness above never asks: is the JSON valid? That is a precondition, checked by the parser, and it costs you nothing. The scoring question is whether the numbers agree with the page. A model that returns malformed JSON is a nuisance. A model that returns clean JSON with a transposed digit is a liability, because nothing downstream will catch it.\n\nIf you want a single headline metric, use silent failure rate: the share of statements where schema validation passes and at least one arithmetic assertion fails. Sort your vendor shortlist on that number.\n\n## Step 5: Separate rounding noise from real error\n\nStatements round display values; the underlying ledger doesn't. A residual of one cent on a 400-row statement is normal. A residual of one cent on a 4-row statement is a misread. So don't use a flat epsilon — log the distribution of residuals per tool and per statement size.\n\nA tool whose residuals cluster at zero with occasional single-cent drift is behaving. A tool whose residuals are bimodal — mostly zero, sometimes wildly off — is guessing on a subset of pages and you need to know which pages. Tag residuals with document type, scan quality, and locale so you can see whether the failures concentrate somewhere you can control.\n\n## Step 6: Score the tools on your numbers\n\nCompare candidates on a scorecard, not a leaderboard:\n\n- Exact pass rate across all six assertions\n- Silent failure rate\n- Row-level field accuracy on amount, date, direction, counterparty\n- Behaviour on scanned vs native PDFs, reported separately\n- Cost per statement and latency per page\n- Determinism: does the same file produce the same output twice?\n- Locale handling: does it get 1.234,56 right without being told?\n\nVendors that read documents and then reconcile against the printed controls before export will score well on the first two lines by construction. That is the design claim behind ZeraBooks, which we make, and it is also why a harness like this one is worth building regardless of which tool you buy — you should be able to verify the claim yourself, with none of the vendor's code in the loop.\n\n## Step 7: Route on the result, not on confidence\n\nModel-reported confidence is not a control. The assertion vector is. Anything that fails a check goes to a human reviewer with the failing assertion attached, so the reviewer knows where to look. Anything that passes all six clears automatically. Tune the auto-clear threshold on your own data, not on the vendor's published accuracy figure.\n\n## Step 8: Re-run on every change\n\nModels are updated silently. Prompts get edited. Rerun the golden set on every model version, prompt change, and document-type expansion, and treat a regression as a release blocker. Every reviewed failure should be added back into the set, so the harness grows toward the statements your business actually sees rather than the ones that demo well.\n\n## Checklist\n\n- [ ] Printed controls captured for every statement in the set\n- [ ] 25+ statements including scans, page breaks, FX lines, and mixed locales\n- [ ] Source file hashes stored with expected values\n- [ ] Decimal arithmetic with explicit rounding, never floats\n- [ ] Six assertions: balance chain, credit total, debit total, row count, period, duplicates\n- [ ] Silent failure rate measured separately from parse failures\n- [ ] Residual distribution logged per tool and statement size\n- [ ] Scanned and native PDFs scored separately\n- [ ] Determinism tested by double-run\n- [ ] Failing statements routed to review with the failing assertion attached\n- [ ] Golden set re-run on every model, prompt, or vendor change\n\n## Sources\n\n- Best AI tools for forensic accountants: testing LLM extraction against printed balances — the original post this harness is based on\n- ZeraBooks: ranking of 9 AI tools for forensic accounting — the vendor comparison referenced above\n- JSON Schema documentation — for the validation layer, and a reminder of what it does and does not prove\n"}



