Experts Review Instead of Retyping: 99% Accuracy in Automated B2B Inquiry Processing

Challenge

The sales department of a heavy-industry customer receives technical inquiries as free-text emails, PDFs, scanned documents, and photos taken on a phone. They arrive in three languages, and input quality varies: tables without borders, empty cells, blurred characters.

A domain expert had to retype every inquiry into an internal form: more than 30 parameters per item, from material designation and chemical composition through mechanical properties to tolerances, standards, and packaging. A typical inquiry has a header with shared requirements and a table of four to eight items. Some requirements apply to every item, others only to a few. Splitting the document into items was manual work, and that is where errors crept in.

Retyping had a second cost. Two experts would record the same inquiry differently, and downstream systems received inconsistent data.

The work took up to five hours a week per expert and sat between an inquiry arriving and a quote going out. Every hour of delay meant a later answer to the customer.

Solution

We built an AI assistant that processes an inquiry in the background and hands the expert a pre-filled form. The user uploads a document, can close the browser, and gets an email when the run finishes. The result takes four steps and is ready within minutes:

1. Split into items. A single model call breaks the document into shared requirements and individual items, reading scans and photos directly. No separate OCR stage.
2. Extract parameter by parameter. Each of the 30+ parameters is extracted by its own call into a fixed schema, together with a citation from the source text.
3. Score confidence. The system computes a confidence score for every value and sorts it into a high, medium, or low band.
4. Review and export. The expert confirms or corrects values in a web interface and exports the result to Excel in one click.

Why a human always approves the output

The system never releases a result without human approval. An item is done only when every extracted parameter is either confirmed by the expert or carries high confidence. An empty extraction cannot be approved at all. In heavy industry, a mistyped tolerance costs orders of magnitude more than a few seconds of review.

99.1% at the parameter level and 60% at the document level are both true statements about the same system. The first says how good the model is. The second says whether it can run without a human.

The gap between the two numbers is not a measurement error. More than 30 parameters times several items adds up to many chances for a single mistake. That is why we chose an assistant with calibrated confidence over full automation.

How it works under the hood

Skip this section if you only care about the impact.

• Dividing the work. Segmentation decides which parameters to extract once for the whole document and which per item, which means 38% fewer model calls on a typical six-item inquiry. Header requirements that apply only to some dimensions are not attached to the other items.
• One extractor per parameter. Each parameter has its own definition, schema, and targeted examples; the schema is the single source of truth for validation and the correction form. Calls run in parallel, and boundaries between neighboring parameters are spelled out so the same value does not land in two fields.
• Confidence outside the model. The model returns evidence, mainly citations from the source text. We verify each citation against the source; if it cannot be found, confidence drops by half. A cap set per parameter limits the result from above. We measure calibration with the band-by-correctness table in the results below.

Learning and operations follow the same logic:

• Learning from corrections. A correction with a rationale is turned into a rule with a general description of the situation it applies to. On the next extraction the system retrieves rules by vector similarity with a high threshold, at most two per call. A poorly matched rule is worse than none. A retrieval failure never breaks an extraction.
• Operations. All calls go through a central gateway that controls concurrency, respects provider limits, and orders calls to make the most of prompt caching. Every call is logged with its cost, duration, and confidence, so we know what one approved inquiry costs.
• No fine-tuning. The model knows the domain. What we added were precise parameter definitions, the boundaries between them, and more than 300 annotated examples.

Results

We measure accuracy on an evaluation set of 50 real inquiries (104 items, 3,432 checked values). Two comparable runs three weeks apart, in May and June 2026, show how far the system moved:

• Parameter-level accuracy rose from 98.8% to 99.1%.
• The number of errors fell from 41 to 30, a 27% drop.
• 75% of items are now entirely error-free (up from 67%), and so are 60% of documents (up from 52%).

The project ships with 231 automated tests on real documents. For every item they check all parameters, not just the expected ones, so a fabricated value counts as a failure automatically. An expert notices a missing value; a fabricated one they would have to hunt down.

Confidence calibration

For day-to-day work, whether the confidence score can be trusted matters more than average accuracy. Values from the June run, by confidence band and correctness:

A value flagged as uncertain is wrong roughly nine times more often than one marked reliable. Experts therefore concentrate on the roughly 12% of values in the low band and confirm the rest. The medium band has a lower error rate than the high band because, for a number of parameters, we deliberately cap confidence below the threshold for confirmation-free approval. It is a conservative setting that we will tighten using production data.

Errors cluster by document. One badly scanned document with four items produces four errors in the same parameter. That is why we also track document-level success, and why the per-parameter error ranking doubles as a prioritized backlog.

For the sales department, work that used to take up to five hours a week now takes minutes of review, and answers reach customers within hours rather than days.

The next step is to skip review for any parameter whose reliable band holds an error rate below 2%. The system is already collecting the data to decide.

What this means for you

The same pattern transfers anywhere people retype unstructured input into a fixed schema: orders, complaints, technical specifications, tender documentation. Principles we carry forward from this project:

• Never let the model grade its own confidence. Have it return evidence and compute confidence outside it.
• Measure accuracy at several levels. The per-parameter number says how good the model is; the per-document number says whether it can run unattended.
• Make negative testing the default. The most dangerous error is not a missing value but an invented one.

Related Reading

• Automating Business Processes with AI Agents — when AI automation works and when it doesn't.
• What Is an AI Hallucination? — why an invented value is worse than a missing one.
• What Is LLM Observability? — what to log to know what every call costs.
• What Is Few-Shot Prompting? — how examples in the prompt stand in for fine-tuning.
• How We Roll Out AI Coding Agents in Large Companies — incremental automation with human oversight at team level.
• Building MCP Governance for Enterprise — governance and audit once AI runs in production.