How we measure agent accuracy

What we measure

Four numbers, not one.

A single accuracy percentage hides the distinction that matters most: the difference between an agent that asks for help and one that is wrong without asking.

Straight-through rate

The share of documents that reach a posted transaction with zero human edit. Reported per workflow, per customer, per week. This is the headline number and the one we publish.

Confidently-wrong rate

Actions taken above the confidence threshold that a person later corrected. The most important number in the system, because these are the errors nobody reviews.

Correction distance

How wrong a wrong answer was. A bill coded to the neighbouring account is a different problem from one coded to a different entity, and averaging them hides both.

Escalation precision

Of the items the agent held for a person, how many genuinely needed one. Escalating everything is trivially safe and useless; this keeps caution honest.

The gate

A change that makes it worse does not ship.

Every prompt and model change runs against a held-out set of your own historical documents before it reaches you. Two gates, both of which must pass.

Why the second gate is the important one

Straight-through rate is the number customers ask about, and on its own it is a dangerous target. You can raise it trivially by lowering the confidence threshold — the agent stops asking for help, more documents post automatically, and the headline metric improves while the system gets materially worse.

That is why confidently-wrong rate is gated separately and why a release that improves the first while degrading the second is blocked rather than traded off. The two numbers move in opposite directions under exactly the optimisation pressure a vendor is under, which is precisely when a hard gate earns its keep.

An agent that asks too often wastes your time. An agent that never asks costs you an audit adjustment. Only one of those failures announces itself.

Where the labels come from

From your team, as a by-product of working. Every correction a controller makes to a proposed coding, every rejected match, every reassigned department becomes a labelled example specific to your chart of accounts and your conventions. Nothing needs to be annotated deliberately, which is why the corpus grows rather than stalling after the initial enthusiasm.

A held-out slice is reserved and never used for retrieval or few-shot context, so evaluation is measured against documents the system has genuinely not been tuned on. This is unglamorous discipline and it is the difference between a real number and a number that has memorised its own test.

Gates are evaluated per tenant

A change that improves accuracy for a distribution business with 400 vendors may degrade it for a consultancy with 30. Because gates run against each customer’s corpus, a release can ship to one tenant and be held for another, and we would rather carry that operational complexity than average across customers and call the mean a result.

What we do not claim

We do not claim a number before we have your data. Any figure quoted in a first meeting is from other customers in a similar shape, and we will say so rather than implying it is a guarantee.
We do not claim high accuracy in week one. The curve below starts near thirty percent. A vendor promising ninety in the first fortnight is describing confidence, not correctness, on a chart of accounts nobody has seen.
We do not claim the reasoning summary is an explanation. It is the model’s account of its own decision, useful for investigation and not a formal proof. The control is the policy evaluation.
We do not claim parity across workflows. Bill coding reaches the low nineties. Vendor deduplication and multi-line allocation are materially harder and sit lower, and we report them separately rather than blending them into one flattering average.

The shape you should expect

Below is a representative curve from a professional-services customer processing around 350 bills a month. Weeks one and two are the agent learning your conventions while every proposal is reviewed. The steep section is the recurring vendor base being learned. The flattening after week sixteen is the long tail — one-off vendors, unusual allocations, and genuinely ambiguous documents that a person should be looking at anyway.

Representative, not guaranteed

This is one real customer’s accounts-payable curve, not a median and not a promise. Volume, vendor concentration, and how consistently your team has coded historically all move it. A business with 40 recurring vendors climbs faster than one with 400 occasional ones.

Questions

On measurement.

Can we see our own numbers?

Yes, in the product, updated weekly per workflow, with the underlying records one click away. They are computed from the audit trail rather than reported separately, so the metric and its evidence are the same data.

What counts as a correction?

Any human edit to a proposed action before or after posting, including a changed account, department, amount, vendor, or a rejection. Approving without modification is not a correction, and reversing an entry later is counted against the original action.

Is our data used to improve other customers agents?

No. Your corrections build your corpus inside your tenant. Improvements that generalise are made to prompts, tooling, and evaluation methodology rather than by pooling customer data.

What if accuracy drops after a release?

The gates are designed to prevent it, but if a regression reaches production it triggers a rollback of the prompt or model version for affected tenants, and the incident is reported to you rather than absorbed quietly.

Do you publish benchmarks across customers?

We publish aggregate ranges by workflow and business type in the research section, with the sample size and method stated. We do not publish a single headline accuracy figure, because it would not mean anything about your business.

How we measure accuracy, and what we refuse to ship

Benchmark it on your documents