The reason most teams under-use AI isn't capability — it's that a naive implementation is too expensive to run on every product, every sync, every request, so it gets rationed until it's not useful. Cost optimization is what makes an AI feature something you run continuously instead of occasionally. Done right, the same enrichment that would cost a fortune per full pass costs about a dollar. Here are the levers, in the order they pay off: caching, change detection, routing, and budgets.
Where the money actually goes
Before optimizing, find the spend. In a commerce AI stack it concentrates in a few places: bulk catalog enrichment (many products times two calls each), the assistant's tool-and-generation loop (multiplied by conversation length), and content generation. The dominant cost driver is almost always redundant work — re-enriching a product that hasn't changed, re-answering an identical request, running a large model where a small one would do. So the first three levers all attack redundancy, and only then do you tune the models themselves.
The order matters because the levers compound multiplicatively, not additively. Caching removes most of the calls; change detection stops the ones caching can't; routing makes the survivors cheaper; budgets cap the tail. Reach for a cheaper model first and you're shaving pennies off calls you shouldn't be making at all. Eliminate the redundant work, and the model-price question shrinks to the handful of calls that genuinely need to happen.
Content-hash caching with a long TTL
The highest-leverage lever by far is caching keyed on content, not time.
We cache AI results on a hash of model + version + input, with a long, durable TTL. If nothing about a product's input changed, the cached enrichment is reused — no model call. Combined with change detection, this is why a full AI pass over a roughly thousand-product catalog costs about a dollar, and why routine syncs are effectively free: almost everything is a cache hit. The cache is durable across runs, so a restart or a new worker doesn't cold-start the bill.
The design points that make content-hash caching work: include the model and prompt version in the key (so a model or prompt upgrade correctly invalidates), make the TTL long because a hash key already captures staleness, and persist the cache so it survives process restarts. This is the economic engine behind treating enrichment as routine rather than a special event (AI product enrichment).
Relevance-aware change detection
Caching only helps if you can cheaply tell what changed — and the subtlety is that not every change matters. A vendor feed re-exporting the same product with a reshuffled field order shouldn't trigger re-enrichment; a real price or description change should. Relevance-aware change detection hashes only the fields that actually affect the output, so cosmetic churn is ignored and genuine change is caught. Get this wrong in the lax direction and you serve stale enrichment; get it wrong in the strict direction and you pay to re-run the catalog every night for nothing. The related discipline of not re-running unchanged work applies to feed ingestion generally.
Route small models first, escalate deliberately
Not every task needs your most capable model. A great deal of commerce AI — classification, normalization, short structured extractions — is handled well by small, cheap models, with escalation to a stronger model reserved for genuinely hard cases. The routing rule is: default to the cheapest model that clears the quality bar for the task, and escalate on a signal (low confidence, a validation failure, an ambiguous input), not by default. The decision of when a bigger model is actually worth it should be measured, not assumed — that's exactly the job of the blind judge in LLM evaluation, which prices the upgrade so you don't pay for capability the shopper never notices.
| Lever | What it saves | Risk if done wrong |
|---|---|---|
| Content-hash caching | Most of a full-catalog pass | Stale output if key omits model/version |
| Relevance-aware change detection | Nightly re-runs of unchanged items | Too lax = stale; too strict = re-runs everything |
| Small-models-first routing | Per-call cost on easy tasks | Quality dip if the bar isn't checked |
| Budgets + rate limits | Runaway loops and abuse | Legit requests capped if set too tight |
Budgets: per-request ledgers and rate limits
Caching and routing reduce the average cost; budgets protect against the tail. Every request carries a cost ledger that attributes spend per feature, so you know which surface is expensive rather than staring at one undifferentiated bill. On top of that: per-session rate limits and a capped tool-call loop, because an assistant that will call tools a fifth time is lost, not thorough, and an uncapped loop is a denial-of-wallet endpoint. Per-feature attribution is what turns "AI is getting expensive" into "the assistant's evidence lookup is the driver" — an actionable statement. These economic guardrails are the same ones in the guardrail chain, viewed through a cost lens.
The cache-poisoning trap
Aggressive caching has one sharp edge worth naming: a cached wrong answer is wrong until you invalidate it.
Early on, a martial-arts uniform was enriched as if it were a supplement — servings, per-serving values, the works — because it was processed before category-aware gating existed. The generated nonsense was then cached, so ordinary re-syncs kept serving the warm, wrong result long after the gating was fixed. Caching had turned a one-time bug into a persistent one. The fix was category-aware profiles plus a deliberate cache bypass to regenerate the poisoned entries — you have to be able to invalidate on purpose, not just wait for the input to change.
So the caching rules carry a caveat: invalidate on model version, prompt version, and classification changes, and keep a manual bypass for when you've fixed a bug upstream of a cached result. Cheap and stale is not a saving. And ship the prompt changes that trigger those invalidations through the canary discipline in prompt management.
Key takeaways
- Attack redundancy first — the dominant LLM cost is re-doing work that didn't need doing, not the price per call.
- Cache on model + version + input with a long durable TTL — it's why a full catalog enrichment pass can cost about a dollar and routine syncs are nearly free.
- Use relevance-aware change detection so cosmetic feed churn is ignored and only genuine changes trigger re-enrichment.
- Route small models first and escalate on a signal, letting a measured judge decide when a bigger model is actually worth it.
- Carry a per-request cost ledger and rate limits so spend is attributable per feature and a runaway loop can't drain the budget.
- Beware cache poisoning — a cached wrong answer persists until you invalidate it, so keep a deliberate bypass for upstream fixes.
Frequently asked questions
How do you reduce LLM costs without hurting quality?
How can a full catalog AI pass cost about a dollar?
Should you always use the most capable model?
What is cache poisoning in an AI pipeline?
AI cheap enough to run always-on.
The caching and budget discipline here is what keeps enrichment and the assistant affordable on the marketplace we operate.
Request early access See the live marketplace →