The AI-in-ERP conversation has moved past "can an agent code an invoice" to a quieter question that controllers are starting to ask on reference calls: where, exactly, does the model run when it reads my general ledger? The answer has three shapes. A shared multi-tenant endpoint from a frontier provider. A dedicated instance inside the vendor's VPC. Or a model running on hardware the customer owns, inside the customer's own network boundary. The default assumption is that only the first option is affordable. That assumption deserves pressure.
Private inference is no longer a research-lab line item. GPU rental has collapsed, open-weight models have closed most of the quality gap for structured finance work, and the regulatory ground under cross-border inference has moved. For a finance team running an AI in ERP deployment at material volume, the per-token API bill and the private-inference bill now sit close enough together that the decision is governed by control, not price. This piece lays out the reasons private AI belongs inside the ledger, and prices out what it actually costs.
Why the Deployment Choice Matters More Than the Model Choice
An AI ERP touches three categories of data the business cannot afford to leak or lose track of: posted journal entries, open vendor and customer records, and the supporting documents that justify both. Every inference call either carries that data to the model or carries the model to the data. The first path, which is what any shared API endpoint does by default, exports the payload across a tenant boundary and often across a jurisdiction. The second keeps it inside a controlled network.
The distinction matters for four reasons that a controller can defend to an auditor.
Residency. SOC 2 does not by itself require data to remain in a single country, but the privacy laws, contractual commitments, and sector regulations layered on top of it frequently do. A shared endpoint can satisfy a residency clause only if the provider offers a named-region deployment and the customer accepts the provider's parent-jurisdiction risk.
Sovereignty. Residency and sovereignty are not the same thing. A region-locked cloud can still fail on sovereignty when the provider's parent faces foreign legal compulsion, and some regulated sectors treat that as a disqualifier regardless of where the bytes sit.
Lineage. Every posted entry needs a traceable path from source document to debit and credit. When the extraction step runs behind a vendor's shared endpoint, the only artifact the customer owns is the output. Private inference lets the organization retain the prompt, the retrieved context, the raw model response, and the deterministic post that followed, as one audit record held in one place.
Model stability. Shared endpoints change under the customer. New model versions drop every six to eight weeks, and the controls a finance team validated last quarter against one checkpoint may behave differently against the next one. A private deployment pins the weights until the team chooses to move.
The AI Risk Management Framework treats all four of these as governance questions, not procurement ones. They tend to surface late in the evaluation, usually when the first external auditor asks where the extraction step ran.
What Private AI Actually Looks Like Inside an ERP
The phrase "on-prem LLM finance" tends to conjure a GPU closet humming in a corner of the data center. That is one shape, but not the most common one. In practice, private AI inside an ERP arrives as one of three deployments.
- Dedicated single-tenant inference inside the vendor's cloud account, with a named region, a signed Data Processing Addendum, and no shared model weights across customers.
- Customer-VPC inference, where the model runs inside the customer's own AWS, Azure, or GCP account and the ERP calls it over private networking. The weights are pinned, the logs stay in the customer's bucket, and no inference traffic crosses the public internet.
- On-premise inference on hardware the customer owns, air-gapped or near-air-gapped. This is the shape mandated under sector rules like HDS, certain PCI DSS interpretations, and some public-sector localization laws.
In all three shapes the architecture is the same: an open-weight model (Llama, Mistral, Qwen, or a finance-tuned derivative) served through a batched inference engine, fronted by a retrieval layer that pulls the relevant ledger slice, vendor master, or document into context at request time. The deterministic ledger the model never writes to directly is a separate concern, covered elsewhere in what an agent cannot do inside a ledger.
The Full Cost Stack for Private Inference
The usual framing pits a monthly API bill against a one-line GPU rental. The honest comparison has more rows. A CFO pricing out a private deployment for an AP agent, an AR agent, and a close-support agent should expect to see six.
Compute. This is the line everyone starts with. An H100 on a competitive cloud rents for roughly $1.99 per hour on PCIe and $2.69 per hour on SXM, with hyperscaler on-demand rates running two to three times higher. H200 instances with 141 GB of HBM3e, which matter for serving 70B-class models at full precision, rent in the $2.50 to $4.50 per hour range on dedicated clouds and above $10 per hour on Azure. A single H100 running Llama-class weights at reasonable batch size costs roughly $1,500 per month in raw GPU time on a continuous basis.
Hardware purchase, when the deployment is on-prem. A standalone H100 PCIe card runs roughly $25,000 to $31,000, with an 8-GPU HGX server at $250,000 to $320,000. An 8-GPU H200 system sits around $370,000 integrated. Most finance deployments do not need an 8-GPU box; a single or dual H100 node is sufficient for an agent fleet serving a mid-market ERP.
Operations. The hidden line. Engineering overhead for vLLM maintenance, monitoring, and capacity planning runs $1,500 to $4,000 per month in engineer time, which is why the break-even analyses that ignore it mislead. Model update cycles alone add roughly $12,000 per year in engineering time for re-testing and re-validation.
Retrieval infrastructure. The vector store, embedding pipeline, and document ingestion layer that put ledger context into the model's hands. Modest in a self-hosted RAG setup, but it is a line, not a free side effect.
Observability and audit logging. Finance workloads need prompt-level, response-level, and tool-call-level logs retained for the same horizon as the journals they influenced. Storage is cheap; the discipline of retaining the right fields is not.
Governance. Someone owns the evaluation suite that catches drift when a model is updated. In a shared-API world the vendor absorbs that; in a private deployment the customer does, which is one reason the AI ERP contract terms matter as much as the architecture.
Against this stack, the API side is simpler but not always cheaper. Frontier API pricing in late 2026 spans a wide range: Claude Sonnet 5 costs $3 per million input tokens and $15 per million output tokens, with Haiku 4.5 at $1 and $5. Open-weight hosted APIs land well below that, with efficient models running in the low single digits per million tokens blended.
When the Math Tips Toward Private
Published break-even analyses from several independent sources converge on a narrow band. Self-hosting beats managed APIs at roughly two to five million tokens per day on reserved GPU capacity over a twelve-month window. Other analyses put the point higher, at 100 million to 500 million tokens per month depending on model tier, GPU pricing, and utilization. The spread reflects which API tier is the baseline and whether the operations line is included honestly.
Three ERP workloads can push a mid-market deployment into that band without heroic volume: document-heavy AP extraction across tens of thousands of invoices a month, continuous reconciliation across multiple entities (the subject of closing faster is mostly a reconciliation problem), and agent-driven anomaly review on every journal line. Combined, they routinely cross the two-to-five-million-tokens-per-day threshold.
Below that threshold, private inference still earns its place when the driver is not price. A customer who requires single-tenancy and runtime residency evidence from every AI vendor is making an architecture decision that no API discount reverses. The same logic applies to firms whose regulators treat the inference call as a data processing event in its own right.
How to Structure the Decision
The CFO question is not "self-host or not." It is which workloads belong on which deployment, held to which controls, and how the erp.io platform wires the two together without the finance team needing to care. A workable sequence:
- Classify each agent workflow by the sensitivity of the data it reads and the materiality of the posting it influences. AP extraction and bank reconciliation sit at the top; narrative drafting for board decks sits lower.
- Measure actual token volume for a representative month, not estimated. The break-even math is useless on guesses.
- Price the stack with all six lines in it. A GPU-only comparison will systematically favor the self-hosted side and mislead the decision.
- Decide the residency and sovereignty constraints before the vendor conversation, not during it.
- Pin the model, pin the evaluation suite, and treat version upgrades as controlled changes the way a schema migration would be.
Private AI inside the ERP is not a status symbol and not a universal answer. It is a specific architectural choice that becomes correct when residency, sovereignty, lineage, or model stability outranks the per-token sticker price, and when sustained inference volume makes the fixed-cost shape rational. For a growing share of mid-market and enterprise finance teams, those conditions now hold together. The remaining work is costing it honestly and contracting for it carefully, which is where the deployment choice stops being a buzzword and starts being a line in the operating budget.