← Back to work

Data engineering · 2026

Retail Sales ETL Pipeline

Production-style batch ETL for a real retail dataset: validation, a star-schema warehouse, Airflow orchestration, and reruns that are safe by design.

incremental re-runs by design
idempotent
schema warehouse model
star
lint + tests on every push
CI

Architecture

Select a stage to explore it.

01 · Extract

Ingests a real public retail transaction dataset — raw exports with the mess left in: cancelled orders, missing customers, inconsistent codes.

pytest + GitHub Actions CIDocker Compose stackPower BI connection guide

Visuals

Retail Sales ETL Pipeline GitHub repository with pipeline architecture
GitHub repository — Airflow-orchestrated batch ETL with star-schema warehouse

01 · Problem

Retail leaders need reliable reporting across time, product category, and market — but raw transaction exports are inconsistent and unsafe to report on directly. The goal: turn them into a governed warehouse model that BI dashboards can trust.

02 · Solution

A batch ETL pipeline with an explicit definition of done: a fresh clone can install dependencies, pass tests, start the Docker stack, trigger the Airflow DAG, and load the warehouse — with incremental reruns that never double-count.

03 · Engineering decisions

Idempotence was treated as a requirement, not a nice-to-have: pipelines fail, and a pipeline you can't safely re-run is a pipeline you can't operate. Validation runs before load so data quality problems surface at the boundary. The star schema trades storage for query clarity — the right trade for a reporting warehouse.

04 · Results

A reproducible warehouse stack with CI running lint and pytest on every push, ready for Power BI on top. The pipeline patterns here — validation gates, incremental loads, safe retries — are the same ones the streaming pipeline later applied in real time.

05 · Lessons learned

Batch ETL is where pipeline discipline is learned: if reruns aren't safe and quality isn't checked at the boundary, everything downstream inherits the problem.