A returns desk might send the same handbook to a model a hundred times before lunch. One customer received a broken kettle. Another has lost a receipt. Each needs an answer specific to the purchase, yet much of the work before the first word is identical. In October 2024 that distinction appeared as a line item on an API bill. It gave a buyer a surprisingly practical question to ask of a prompt: which part of this request have we already paid to process?
What survives between two requests
On 1 October 2024, OpenAI began charging half the ordinary input price for eligible cached tokens on selected models. Its launch announcement described reuse of the longest previously computed opening sequence, starting at 1,024 tokens and increasing in blocks of 128. A token is a piece of text in the model’s input. A phrase that expresses the same policy in slightly different words can produce a different token sequence.
Imagine the first line of every prompt carries a ticket number. Two tickets now diverge almost immediately, even if the next ten pages match word for word. Move the variable ticket details below the shared instructions and handbook, within the appropriate message roles, and the common opening becomes eligible for reuse. The system still has to process the new details and generate the reply. The second customer can receive a different answer even when almost all the input was cached.
The difference matters for conversational software. A cache of completed answers needs a rule for deciding when an earlier answer remains applicable. Here the model sees the full new request, including the customer's new question. OpenAI’s prompt caching cookbook also illustrates reuse with tool definitions and conversation history. A team can accidentally disturb the shared opening by reordering its tools on every call, even when nobody has edited the policy.
Read the invoice for a hundred tickets
Consider a hypothetical morning with 100 requests. Each contains 4,000 input tokens and returns 250 output tokens. Assume 75 requests each report 3,072 cached tokens. The other 25 report none. For the August 2024 GPT-4o snapshot, the October announcement listed US$2.50 per million uncached input tokens, US$1.25 for cached input and US$10 for output. Those launch prices let us reconstruct the bill in historical dollars.
Scroll horizontally to compare the figures.
| Billable work | Tokens with reuse | Without reuse | With reuse |
|---|---|---|---|
| Uncached input | 169,600 | US$1.000 | US$0.424 |
| Cached input | 230,400 | US$0.000 | US$0.288 |
| Output | 25,000 | US$0.250 | US$0.250 |
| Total | 425,000 | US$1.250 | US$0.962 |
The 75 hits account for 230,400 cached tokens, or 57.6% of the 400,000 input tokens. Charging those tokens at half price saves 28.8% of the input bill. Output adds another US$0.25 in either case, so the saving on the combined bill is about 23%. All three percentages answer different questions. “Seventy-five per cent of calls hit the cache” is accurate in this example, but a poor number to give the finance team on its own.
Longer answers change the result. If every customer asks for a detailed letter and the replies grow to 1,000 tokens each, the uncached total rises to US$2.00. The same US$0.288 saving is now 14.4% of the total. No engineering change occurred between those two estimates. The output mix alone changed what the buyer could reasonably expect to save.
When a cache miss is the right result
Suppose the handbook changes at noon. A promotional return window now applies to kettles sold in December. Someone might suggest moving the amendment to the end of a long prompt to preserve the discount. If the old rule still appears as an unconditional instruction near the beginning, the model must reconcile conflicting directions. Publish one authoritative version of the policy and accept the cold requests associated with the change. Test one purchase on each side of the effective date.
The cache also rewards traffic that recurs soon enough. The launch announcement said entries typically cleared after five to ten idle minutes and expired within an hour of last use. A busy returns desk and a small wholesale queue could share a handbook template and achieve quite different savings. Likewise a retrieval system that fetches three different clauses per customer may legitimately have a short common opening. Padding every request with boilerplate to increase the hit rate can increase the total amount paid.
Replay the queue before changing the budget
Take a representative week of ticket arrivals and preserve their time intervals in a replay. Include the first request of the day, the afternoon burst and a policy revision. Run a baseline with the current prompt order and a candidate with stable instructions first. Use distinct test prefixes of comparable length for the two arms so warming one does not warm the other. Keep the model snapshot and output limit fixed. Feed both versions the same tickets in the same order.
For each request, capture total input, reported cached input, output, policy version and elapsed time to first word. The usage response is sufficient to count tokens without putting customers’ names or entire prompts into a billing report. Sum token volumes across the week, apply the relevant prices and divide by the number of correct, complete responses. Inspect a sample of answers against the handbook, including the two boundary cases. An answer grounded in an expired policy has no value to the desk however little it cost.
For teams evaluating model access through Cooma Model API, ask which model and route the quotation covers, how the provider reports cached input and what prices apply on that route. Carry the measured volumes into the quote. The returns manager can then see both the likely bill for the next hundred tickets and how much it depends on customers arriving close together.