Most AI ERP pilots fail the audit committee before they start. Not because the software misbehaved, but because the pilot was scoped to demonstrate capability rather than to produce evidence. A demo of an AP agent posting an invoice is not proof that the agent can be trusted with the ledger. Proof requires a defined workflow, a defined control, a measured baseline, and a rule that turns the numbers into a keep or stop decision.
Two weeks is enough time to produce that evidence, if the scope is narrow and the instrumentation is set before kickoff. Two months is not enough time if it is not. What follows is how to structure the paid pilot so the answer you carry back is defensible.
Pick One Workflow, End to End
The mistake is to pilot "AP" or "the close." Both are too broad to measure. Pick a single workflow, defined by its entry event and its exit event, and run it end to end inside the pilot tenant. A workable AP agent evaluation looks like this: PDF invoice arrives in the shared mailbox, agent extracts header and line data, matches to the PO and receipt, codes the GL and cost center, routes for approval where policy requires it, and posts to the ledger. Entry: email received. Exit: journal entry with a link back to the source document.
A workable cash application scope: bank file lands, agent proposes customer and invoice matches with a confidence score, controller reviews anything under threshold, remainder posts. A workable close scope: one legal entity, three reconciliations (bank, intercompany, prepaid), fully agent-run, controller signs off. Pick one. Not three.
Narrowing scope is what makes the pilot honest. The published advice on avoiding pilot purgatory is to scope one workflow, commit to CFO-grade KPIs, embed governance and evidence, and define graduation criteria before kickoff. If the vendor pushes to add scope during the two weeks, the pilot has already become a demo. This is also where the approval threshold work matters: the pilot has to run under the same policy that would apply in production, not a relaxed one.
Baseline Before You Start, Not After
An AI ERP proof of concept without a baseline is a story about the vendor. A pilot with a baseline is a comparison. Before the two weeks begin, pull the last quarter's numbers for the same workflow: cost per invoice, cycle time from receipt to post, touchless rate, exception rate, and the count of invoices that required a correcting journal after the fact.
The published benchmarks give you a defensible ceiling to test against. Top-performing AP teams process invoices in 3.1 days at $2.78 each with exception rates of 9% or lower, against an industry that averages $12.88 and 17.4 days. Touchless rate is the metric that pulls the rest along: the 2026 top-quartile benchmark is above 80%, while most teams on legacy AP automation sit between 30% and 50%. Write your current numbers down. Sign them. That is the "before" the pilot has to beat.
One caveat on the industry figures: the most common business-case error is overstating achievable touchless rate by using vendor best-case numbers rather than production estimates on the actual document corpus. The pilot exists to prove the vendor's number on your invoices, not the other way around.
Instrument the Controls, Not Just the Outputs
The audit committee does not care that the agent posted 400 invoices. They care what the agent could and could not do, what was logged, and what would have stopped a bad post. Every pilot should instrument four control layers, and the vendor should surface them in a read-only dashboard the controller can pull mid-pilot.
- Segregation. The agent proposes journal entries; a deterministic ledger accepts or rejects them. This is the structural line that separates a governable AI ERP from a wrapper. The site's own piece on what an agent cannot do inside a ledger is the frame to hold vendors to.
- Traceability. Every posted entry links back to source document, extraction confidence, coding rationale, and the policy rule that authorized it. Pull ten random entries. If the trail is not one click, the control is not there.
- Reconciliation proof. Sub-ledger to GL reconciliation runs daily during the pilot, not at the end. The controller should see the reconciliation report on day three, not day fourteen.
- Override logging. Every human correction of an agent decision is captured with reason code. A high override rate signals a mismatch between the model's behavior and the policy it is meant to follow, and that signal is invisible if overrides are not logged separately from ordinary edits.
The NIST AI Risk Management Framework is the reference the audit committee will recognize if you need to justify the control set in language a board understands.
Set the Thresholds Before You See the Numbers
Exit criteria written after the pilot are not exit criteria. They are rationalizations. Write them into the statement of work, signed by finance and by the vendor, before day one.
A defensible threshold set for an AP pilot, calibrated to published benchmarks:
- Extraction accuracy on header fields (vendor, invoice number, date, amount, currency): 99% or higher on a sample of at least 200 invoices spanning your ten most variable suppliers.
- GL coding accuracy against the controller's post-hoc review: 95% or higher.
- Touchless rate on eligible invoices: 70% or higher during the pilot, with a plan to reach 80% inside one quarter.
- Exception rate: below 15%, with every exception traceable to a named reason (missing PO, price variance over tolerance, new vendor, tax mismatch).
- Zero unreconciled sub-ledger to GL differences at the end of any pilot day. Not "small." Zero.
- Zero posted entries without a linked source document and policy citation.
Two of those are hard gates. A single unreconciled day, or a single posted entry without provenance, is a fail regardless of how well the other numbers landed. That is what makes the pilot defensible: the fail condition is written down, and it is not negotiable.
Send the Ugly Invoices First
Vendors will happily process your clean PO-matched invoices from your top ten suppliers. The pilot has to test the tail. Load the 200-invoice sample with the ratio of edge cases the production system will actually see: non-PO invoices, foreign currency, credit memos, multi-line freight, utility bills with meter reads, invoices with handwritten annotations, duplicates that are not exact duplicates.
The standing advice on benchmark testing is direct: prove each row against your own historical invoices, not the vendor's sample set, and send the ugly ones first. This is also where the bad data that surfaces during load becomes evidence rather than a surprise.
Define What a Fail Looks Like
Most pilots produce ambiguous results because "fail" was never defined. Three categories of fail are worth writing into the SOW explicitly, each with a different consequence.
Structural fail. The agent writes directly to the ledger without going through a deterministic accept-or-reject layer. Or the source document trail is broken. Or override logging is not distinguishable from normal edits. These are not tunable. If any is true, the pilot ends and the vendor does not proceed to contract. The deterministic ledger architecture the pilot is supposed to prove either exists or it does not.
Performance fail. The accuracy or touchless numbers miss the thresholds by more than a defined margin (say, 5 percentage points on any single metric, or any two metrics missed at all). This one is negotiable: extend the pilot by two weeks with a tuning plan, or stop. Do not sign.
Operational fail. The pilot hit the numbers but the controller cannot explain to an auditor what the agent did on a specific invoice within five minutes. This is a documentation and observability gap, not a model gap, but the ledger you inherit will carry it forever. Fix before signing or walk.
The vendor contract is the next decision after the pilot, and the contract terms worth negotiating are easier to hold when the pilot produced numbered exit-criteria evidence rather than a slide deck.
What to Carry to the Audit Committee
The deliverable at the end of two weeks is not a recommendation. It is a packet: the signed baseline, the signed thresholds, the measured results against each threshold, the exception log with reason codes, the daily reconciliation reports, a sample of ten agent-posted entries traced end to end, and the override log. The recommendation follows from the packet, and anyone on the committee can read it in the same order and reach the same conclusion.
A pilot structured this way answers the only question the committee actually has, which is not whether the AI works. It is whether the controls around the AI are strong enough that a bad model day would be caught before it reached the trial balance. If the packet shows that, the signature is defensible. If it does not, the two weeks were still worth running, because the alternative was finding out in production.