A product feed is a firehose that never turns off. The vendor changes a price, restocks a size, discontinues a flavour, adds a hundred products, removes fifty — and your marketplace has to absorb all of it, every sync, without double-charging for work, leaving discontinued products visible, or falling over when a feed is briefly malformed. Reliable ingestion is less about parsing and more about discipline: knowing what changed, doing only that, cleaning up what left, and never trusting the feed to be well-formed. Here is the shape of ingestion we run in production against live vendor stores.
Full syncs and partial syncs
Two cadences do most of the work, and they exist for different failure modes.
| Full sync | Partial sync | |
|---|---|---|
| Cadence | Nightly | Hourly |
| Reads | The entire catalog | Only products flagged as changed |
| Catches | New products, deletions, restructured categories | Price and stock movement |
| Role | Reconciling safety net | Keeping fast fields fresh cheaply |
The full sync is authoritative: it is the run that can add, update and remove, so it is the only place a deletion is ever discovered. The partial keeps the day-to-day fields current without re-reading everything. If you only build one, build the full sync — a marketplace that never reconciles accumulates lies.
Idempotency: run it twice, get the same catalog
The single property that makes ingestion safe to automate is idempotency: running the same sync twice produces the same catalog as running it once. This is what lets you retry a failed run without fear, run a full and a partial that overlap, or re-ingest after a deploy. Achieve it by keying every product on a stable identifier from the source and upserting — insert or update by key — rather than appending. A pipeline that appends will duplicate a vendor's whole catalog the first time a run is retried.
Idempotency has a twin: resilience to a briefly malformed feed. A sync should treat a feed that suddenly fails to parse, or returns implausibly little, as a reason to abort and keep the last good catalog — not as an instruction to wipe everything. The safe default when a run looks wrong is to change nothing and alarm, because a marketplace that trusts a broken feed will faithfully delete its whole catalog and only notice when the traffic disappears.
Idempotency is also what makes the enrichment layer affordable, because it lets you cache aggressively: the same product with the same data can be recognised across runs and skipped. That recognition is the next piece.
Content-hash change detection
Naive ingestion re-processes every product on every sync, which is fine until enrichment enters the picture — then you are paying to re-generate copy for products that did not change. The fix is change detection keyed on content.
Every product carries a content hash over the fields that actually matter to enrichment. On each sync we compare the incoming hash to the stored one; if it matches, the product is skipped entirely — no re-enrichment, no re-index. The subtlety that earns its keep is relevance-aware hashing: a price or stock change moves fast-field data but touches nothing a shopper reads, so it does not invalidate the hash and does not trigger expensive regeneration. One caveat we learned to respect: the hash covers top-level content, so a change confined to a single variation has to be included deliberately or it will be skipped.
This is the mechanism behind the enrichment economics we describe in AI product enrichment — a full AI pass over a roughly thousand-product catalog that costs about a dollar, because routine syncs skip almost everything. Change detection is not an optimisation you add later; it is what makes an always-on feed financially possible.
Orphan cleanup when a catalog shrinks
Growth is easy; shrinkage is where pipelines rot. When a vendor removes fifty products, a naive per-vendor sync that only upserts what it sees will leave those fifty as orphans — present in your search index, your vector store and your database, absent from the vendor's feed, invisible to the vendor but very visible to a shopper who buys one. Removing a product is a three-place delete: the keyword index, the vector index, and the primary store, all of which must be reconciled against what the full sync actually saw.
The rule: a full sync must compute the difference between "what the vendor has now" and "what we have", and remove the leftovers everywhere. Skipping this produces ghost products, and ghost products are a direct hit to trust — the same reason offboarding a whole vendor is a careful, ordered purge rather than a flag flip, covered in vendor offboarding.
Gating imageless products at ingest
Not every product that arrives should be listed. The clearest example is the imageless product.
Products that arrive with no image are dropped at ingest, before they ever reach the index. A grey placeholder in a grid of real photos reads as a broken or abandoned catalog, and no amount of good copy compensates for it. Gating at ingest — rather than hiding downstream — keeps the defect out of every surface at once: search, browse, recommendations. It is one deterministic check, applied in one place, that a whole class of ugliness never gets past.
Imageless is the sharp example, but the principle generalises: ingest is the right place to enforce hard quality gates, because a defect stopped there cannot leak into any downstream surface. Which defects deserve a gate is the subject of product data quality.
Monitoring ingest health
An always-on pipeline needs to tell you when it is unwell, and the signals are cheap to emit: how many products were seen, added, updated and removed on each run; how long the run took; how many were gated or quarantined. The alarms that matter are the anomalies — a sync that removed 90% of a vendor's catalog is far more likely a broken feed than a real fire sale, and you want to catch it before it reaches shoppers, not after. Keep the metrics bounded and free-tier-friendly; the discipline that keeps observability from quietly becoming expensive is its own lesson in observability for a small team.
Key takeaways
- Run a nightly full sync and an hourly partial; only the full sync can discover deletions, so it is the one you build first.
- Make ingestion idempotent — upsert on a stable key so a retried run cannot duplicate a catalog.
- Detect change on a content hash, relevance-aware, so price and stock updates do not trigger expensive re-enrichment.
- A shrinking catalog needs a three-place delete — keyword index, vector store and database — or you accumulate ghost products.
- Gate hard defects at ingest; dropping imageless products in one place keeps the problem out of every downstream surface.
- Emit bounded health metrics and alarm on anomalies like a sync that removes most of a catalog — usually a broken feed, not a sale.
Frequently asked questions
What is the difference between a full and partial product sync?
How do you avoid re-enriching products that didn't change?
What are orphan products in feed ingestion?
Should products without images be listed?
How do you monitor whether ingestion is healthy?
An always-on catalog that stays honest.
Our managed marketplace ingests vendor feeds on a schedule with change detection and cleanup built in, so the catalog never drifts or fills with ghosts.
Request early access See the live marketplace →