Held-out evaluation sets
Several thousand real cases with known correct outcomes per workflow, never used in prompt development, so a score measures generalisation rather than memorisation.
AI capability
Every vendor claims their agents are accurate. Almost none will tell you what they measured, on how many cases, against what ground truth, or what happens when a release makes it worse. Those four questions separate a measured system from a confident one.
What it does
Several thousand real cases with known correct outcomes per workflow, never used in prompt development, so a score measures generalisation rather than memorisation.
An agent claiming 95% confidence should be right about 95% of the time. We measure whether it is and adjust the thresholds rather than the claim.
Any release that regresses accuracy or calibration on the evaluation set does not ship. Not a warning, not a review item — it does not ship.
Deliberately hard examples — near-duplicate vendors, invoices with two plausible codings, documents with conflicting totals — held separately because averages hide the cases that cause damage.
Reversal rate, override rate, and escalation rate tracked per customer, because a set that was representative last year may not describe your business now.
Our benchmarks page shows the median and the interquartile spread. The bottom quartile is materially worse than the median and we print it.
What did you measure. On how many cases. Against what ground truth. What happens to a release that scores worse. Ask those in any AI-in-ERP evaluation and the range of answers is informative — many vendors cannot answer the second, and very few can answer the fourth.
The fourth is the one that matters most, because it is the difference between measuring and being governed by measurement. An accuracy number with no gate behind it is a marketing figure that happens to be true on the day it was produced.
The most common way an evaluation becomes meaningless is quiet contamination: cases used during prompt development end up in the set, and the score measures how well the agent has memorised examples somebody tuned against.
Our evaluation sets are constructed before prompt work and never examined during it. Failures are analysed in aggregate — categories and rates, not individual cases — precisely so the set stays clean. It is a real constraint and it is the only way the number means anything.
An agent that is right ninety percent of the time and claims certainty on all of it is far more dangerous than one that is right eighty-five percent of the time and knows which fifteen it should not act on.
The second can be given authority safely, because the threshold does the work. The first cannot be given authority at all, because its errors are indistinguishable from its successes until somebody finds them. We measure both and treat calibration as the more important property.
Averages flatter. A workflow at ninety-one percent overall might be at ninety-eight on recurring vendors and sixty on first-time ones — and first-time vendors are where the fraud risk and the coding errors concentrate.
So hard cases are held in their own set and scored separately, and the authority thresholds reflect the hard-case performance rather than the blended one.
The most useful evaluation is the one run on your transactions. Send several hundred historical cases with known outcomes and we will score against them and show you the result, including where it does poorly.
It is the only benchmark that predicts your experience, and a vendor unwilling to be measured that way before a purchase is telling you something.
Limits
Every capability page on this site carries one of these, because a feature described without its boundaries is a claim rather than a description.
A benchmark from other customers predicts your result imperfectly. That is why we offer a test against your own historical transactions before you commit.
For some decisions two experienced accountants disagree. Where that is true we say the ceiling is bounded by that disagreement rather than by the agent.
A failure mode absent from the set is a failure mode untested. Production monitoring exists because evaluation is necessary and not sufficient.
Questions
What was measured, on how many cases, against what truth, and what happens when a release scores worse.