Secure file intake
Daily inventory files from many suppliers are collected on schedule into one place - no manual gathering.
Work / Flagship case study
Deep dive - InventoryMillions of stock rows, read at a glance every morning.
Supplier inventory feeds change overnight in ways nobody has time to read line by line. This system watches what moved - and surfaces only the changes that matter for stock decisions.
DailyScheduled ingest of supplier feeds
MillionsRows compared in each snapshot
InstantDashboard reads a cached overview
Delta onlySurfaces day-over-day changes
Inventory truth lived across many supplier feeds, each refreshed on its own schedule and totalling millions of rows. Somewhere in that flood, items went out of stock, came back, moved sharply, or disappeared from a feed entirely - and any of those could quietly cost a sale or strand a listing. Reading the raw files was impossible; loading them naively made a dashboard crawl.
What a buyer actually needs isn't the whole snapshot - it's the difference from yesterday, ranked by consequence. So the system was built to compute that difference reliably at scale, and to make the answer appear instantly rather than recompute on every page load.
The approach
01
02
03
04
05
06
Compare snapshots once, serve the answer instantly.
Heavy comparison happens on a schedule and gets cached; the dashboard only ever reads a small, precomputed overview - so it stays fast no matter how large the underlying data grows.
pandas ingest
Each feed is melted into tidy per-warehouse rows and normalized into a multi-million-row snapshot store.
Snapshot compare
Today's snapshot is diffed against yesterday's to find out-of-stock, back-in-stock, significant moves, and dropped items.
Availability rules
Explicit thresholds classify each change by stock risk, so a big drop reads differently from ordinary drift.
Overview cache
A compact summary is precomputed and stored, so the dashboard never has to scan millions of rows live.
Dashboard & exports
A lightweight dashboard shows visibility at a glance and produces cleaned, marketplace-ready upload files.
Stack: Python - APIs - data processing - SQL database - web dashboard - secure file intake - scheduled jobs.
What a morning's change report looks like.
A representative slice of the day-over-day overview the dashboard surfaces - the same shape a buyer would scan first thing. Everything below is synthetic sample data; no supplier or client information is shown.
| Item ID | Warehouse | Qty yesterday | Qty today | Change | Signal |
|---|---|---|---|---|---|
| ALT-8890 | West - CA | 112 | 0 | -112 | Out of stock |
| BRK-1042 | Central - TX | 840 | 190 | -650 | Significant drop |
| RAD-5567 | East - NJ | 0 | 64 | +64 | Back in stock |
| WPR-1120 | West - CA | 1,204 | - | - | Dropped from feed |
| SPK-3301 | Central - TX | 2,900 | 3,010 | +110 | Steady |
| BAT-9004 | East - NJ | 58 | 6 | -52 | Significant drop |
Only rows that crossed a threshold appear here - the millions of unchanged rows never reach the buyer's screen.
Why it holds up
The choices that keep it fast and honest.
01
Precompute, then serve
The expensive comparison runs once per cycle and is cached, so dashboard response time stays flat as the data grows.
02
Explicit thresholds
What counts as a "significant" move is a stated rule, not a hunch - so signals mean the same thing every day.
03
Dropped items are a signal
Items that vanish from a feed are surfaced, not silently ignored - a missing row is often the most important change.
04
Clean exports downstream
The same pipeline emits marketplace-ready files, so visibility and action come from one source of truth.
Next case study
Read it ->
A pricing pipeline that only ever changes what it's sure about.
How the nightly Pricing Update Pipeline fails safe on cost columns and ships the smallest safe change set.
Let's talk
Want me to walk you through the decisions behind it?
I'm happy to go deeper on the architecture, the rules, or the trade-offs. Grab a time or send a note.