The citation is real. Does it support the answer?

A contract assistant can quote the right clause and still give the wrong answer. A worked review queue shows when improving the evidence saves more than changing models.

A customer cancels three days before delivery. An assistant returns “The deposit is refundable” and links to a contract paragraph containing those exact words. The reviewer opens the paragraph. Its next sentence limits refunds to cancellations received at least seven days before delivery. The retrieval system found the relevant contract. The citation points to the right passage. What failed was the inference that connected the passage to the customer’s dates.

In January 2025, Anthropic added citations to its API. Its document features can associate answer text with locations in source material supplied by the application. That gives a reviewer a much shorter route back to the evidence. It also gives an enterprise a way to examine where a document assistant breaks down. Buyers can separately test the material supplied to the model, the support for each claim and the human time spent deciding whether to accept the answer. Anthropic’s January release notes and citations documentation.

Document analysis Checking the pointer establishes where a passage came from. Checking the exception establishes whether it applies.

Preserve the evidence before measuring the model

Start with the document as received. Record the agreement identifier, revision, effective date and a stable reference to the original. A replacement file with the same name can change the answer without changing the question. Where an application extracts text from a scanned page, keep the original visible to the reviewer. A missing minus sign or displaced table heading can turn a faithfully quoted extraction into an incorrect decision.

Next record which passages reached the model. For a refund question the answer may depend on a general clause, an exception several pages later and a dated amendment that supersedes both. If the amendment never reaches the model, the team needs to revisit retrieval or document selection. If the amendment reaches the model and it still quotes the older clause, compare reasoning and instruction following. Those failures have different owners. A larger context window helps only if the application includes the right records within it.

Finally, bind each factual claim in the displayed answer to its reference. “You qualify for a refund of the deposit and delivery charge” contains at least two claims. One paragraph may settle the deposit and say nothing about delivery charges. In their ALCE research, Gao and colleagues distinguished whether citations support an answer and whether statements have adequate citation coverage. That distinction is directly useful when deciding what an assessor should mark on a review sheet. Enabling Large Language Models to Generate Text with Citations.

Change one fact at a time

Write the expected conclusions with a policy owner before choosing the model. For the example below, assume the signed terms say only that deposits are refundable with at least seven days’ notice. Present the same policy text in every case. Vary the dates and the documents in a controlled order so that a failed case points to a particular weakness. These examples illustrate how to construct test inputs. They do not describe a customer agreement.

Scroll to see all columns.

Acceptance cases for one cancellation clause
Evidence suppliedExpected treatmentError the test exposes
Ten days’ notice and a confirmed delivery dateApply the refund clause and cite the notice requirement.The assistant reaches the right outcome for the wrong reason.
Three days’ notice and the same agreementExplain why this clause gives no entitlement to a refund.The answer quotes the general rule but omits its condition.
Cancellation date onlyAsk for the delivery date before deciding.The model fills in a fact absent from the source.
Three days’ notice and a signed two-day amendmentApply the amendment if it governs this booking and cite its link to the agreement.Retrieval or reasoning favours the superseded version.

Account for the answers a reviewer has to repair

Imagine an evaluation of 100 cases. The reviewers identify 80 that can be answered from the supplied material and 20 that require a clarification. Suppose the assistant gives supported, correct answers on 72 of the 80 and correctly asks for more information in 15 of the 20. That leaves eight deficient answers on resolvable cases and five unjustified decisions on cases with missing evidence. A report showing 87 acceptable outcomes out of 100 hides that last group. A policy owner may tolerate a delay more readily than an unsupported commitment to a customer.

Now attach costs to the same cases. Assume, for illustration, A$12 of model and retrieval costs, one minute of review per case and five additional minutes for each of the 13 deficient outcomes. With reviewer time valued at A$60 per hour, the queue consumes 165 minutes and A$177 in total. If every deficient outcome is resolved in those extra minutes, the resulting cost is A$1.77 per closed case. If some remain open, count only the closures in the denominator and keep the pending work visible.

That calculation changes the model comparison. Suppose a second configuration costs A$20 for the same queue and reduces deficient outcomes to seven, with the same base review time. It would cost A$155 in total under those assumptions. The extra A$8 of inference pays for itself once it avoids more than 1.6 five-minute corrections. In a pilot, measure actual correction time by failure type. Finding a missing delivery date and unwinding a refund already promised by email rarely take the same effort.

Decide where to put the next dollar

When the correct passages are absent, try a retrieval change before paying for a more expensive generator. When the right evidence is consistently present, compare models on the same frozen inputs. Keep the reviewer blind to model identity, rotate case order, and keep related amendments together when splitting development and final test sets. Otherwise a team can appear to improve by rehearsing near duplicates of cases it has already tuned.

Require the supplier to demonstrate the selected model, endpoint and document types with your own sample. Ask how a reference behaves after a file is replaced, what happens to image-only pages and whether access to the underlying evidence follows the reviewer’s permissions. Cooma Model API provides project-level model access, limits and spend visibility. For a citation-sensitive workflow, verify the precise provider feature and returned citation fields for that project. A general model access agreement alone cannot tell a reviewer where a given answer came from.

The delivery team can then take a precise improvement request to the right owner. “Nine amendments were missing from retrieval” calls for different work from “reviewers spent two minutes opening each reference.” Each can be tested on the same cases and valued in minutes and accepted decisions.