Across 23 customers we measure our agents at 94% on their strongest workflow and 66% on their weakest. Both numbers, the sample and its bias are on the benchmarks page; 66% is the one worth reading. This post is about why the gap exists and why we do not expect it to close by making the model better.
The distinction that predicts the number
A task is a good fit for an agent inside a financial system when a wrong answer is detectable from inside that system. It is a bad fit when the only thing that can tell you the answer is wrong is a human who knows something the system does not contain.
Matching a payment to an invoice is checkable: the amounts either reconcile or they do not, the reference either resolves or it does not, and a wrong match leaves a residual that something downstream will notice. Coding an ambiguous expense to a cost centre is not checkable. Both cost centres are valid. The journal balances either way. Nothing anywhere in the system will ever object, and the error is discovered — if it ever is — by a budget holder in a review meeting six weeks later.
The agent is not less intelligent on the second task. It is being asked a question the system cannot mark.
That is why the accuracy ordering across our workflows tracks checkability rather than difficulty. Reconciliation, which involves more steps, scores far higher than classification, which involves one.
What this means in practice
The design consequence is that autonomy should be set per task by checkability, not per customer by confidence, and certainly not globally by a slider in a settings screen.
- Checkable and high-volume: let it run, and reconcile. The control is not review, which does not scale and which people stop doing. The control is that a wrong answer produces a discrepancy something else will catch.
- Uncheckable: propose, never post. The agent’s useful output here is a ranked suggestion and its reasoning, not an entry. A 66% workflow that proposes is genuinely useful. The same workflow posting unsupervised is a slow corruption of the data that everything downstream depends on.
- Anything that moves money outside the company: a person, every time. Not because the accuracy is bad, but because the recovery cost is asymmetric and no accuracy figure makes it symmetric.
The part vendors skip
Every agent in a ledger writes. That is the point of it. Which means the honest question about any agent feature is not what it does when it is right — it is what the trace looks like when it is wrong.
Three things we would want to see demonstrated in any product, ours included, before an agent is allowed to post:
- Every agent action ties back to a source record, and the tie is queryable rather than logged as free text.
- A wrong action is reversible by a documented mechanism, not by a support ticket and someone with database access.
- The accuracy figure quoted was measured on the customer’s own transactions, not on a vendor benchmark, because the distribution of ambiguous cases is exactly what differs between businesses.
What we get wrong
We have shipped agent behaviour that was too eager. The classification workflow now sitting at 66% originally posted rather than proposed, and the correction that followed is in the changelog with the previous behaviour stated. That is the right way round: an accuracy number attached to a task that should not have been autonomous is a number describing the wrong thing.
Our position on what agents may do at each autonomy level is on the AI pages, and the measured figures — including the spread, not just the median — are on the benchmarks page.