Python · pandas · openpyxl · Next.js
PIM Pipeline
Python pipeline that normalizes 10,000+ product attributes across systems
10,000+
product attributes normalized
60%
manual data entry cut
8 hrs/wk
manual work eliminated
Problem
Product data lived in two source systems that disagreed constantly — different headers, formats, and update cadences across 10,000+ attributes. Keeping storefront catalog data correct meant roughly 8 hours a week of manual spreadsheet reconciliation, and errors that slipped through surfaced directly on live product pages.
Approach
Built a pandas pipeline that ingests both sources (openpyxl for the spreadsheet side) and runs header-diff comparison to detect schema drift before processing a single row.
Normalized attribute names, formats, and units into one canonical schema — a single source of truth for downstream systems.
Made every write an idempotent upsert, so re-running the pipeline after a partial failure is always safe.
Shipped a Next.js review dashboard that surfaces diffs for human approval, keeping a person in the loop for ambiguous changes instead of silently overwriting.
Architecture
+------------+ +------------+
| Source A | | Source B |
| (exports) | | (exports) |
+-----+------+ +-----+------+
+--------+---------+
v
+------------------------------+
| Python / pandas pipeline |
| header-diff comparison |
| normalize -> validate |
| idempotent upserts |
+---------------+--------------+
v
+------------------------------+
| Next.js review dashboard |
| approve / reject diffs |
+---------------+--------------+
v
+------------------------------+
| Storefront catalog data |
+------------------------------+Results
8 hrs/week of manual data-entry work eliminated — a 60% reduction.
10,000+ product attributes normalized across two source systems into one canonical schema.
Downstream attribute errors on live product pages reduced, with every change reviewable in the dashboard before it ships.
Idempotent upserts make the pipeline safely re-runnable after any partial failure.