Real-time data · 2026
Streaming Clickstream Pipeline
A real-time lakehouse for e-commerce clickstream — Kafka to Spark to Delta Lake to a live dashboard. Seven services, one command, exactly-once semantics.
- events ingested
- ~50/s
- micro-batch cadence
- 10 s
- services, one command
- 7
Architecture
Select a stage to explore it.
01 · Event generator
Synthesizes ~50 clickstream events per second — page views, searches, cart actions, purchases — including deliberately invalid messages to exercise the dead-letter path.
Visuals

01 · Problem
Batch pipelines answer yesterday's questions. E-commerce platforms emit a continuous stream of behavioral events, and understanding conversion funnels or product performance in real time means ingesting, processing, and serving those events with minimal latency — without losing any when things restart.
02 · Solution
An event-driven pipeline that moves a click from the browser to an analytics dashboard in under a minute: Kafka for durable ingestion, Spark Structured Streaming for processing, a Delta Lake medallion architecture on S3-compatible storage, DuckDB for serving, and Streamlit for the live view. The whole stack starts with one command.
03 · Engineering decisions
Delta Lake over raw Parquet: ACID transactions and schema evolution are worth the metadata overhead. DuckDB over a warehouse: zero-config and fast, accepting no multi-user concurrency — right-sized for the workload. One foreachBatch query for all three layers: simpler checkpoint management, at the cost of a shared processing cadence. MinIO over cloud S3: fully reproducible local development, one config change away from AWS.
04 · Challenges & trade-offs
The hard part of streaming is failure, not throughput. The pipeline achieves effectively-exactly-once semantics through idempotent Delta writes plus Kafka offset tracking, recovers from restarts via checkpointing and consumer-group rebalancing, and handles out-of-order events with watermarking — while a 7-day range filter drops stale replays.
05 · Results
Sub-minute latency from raw event to dashboard, with invalid events quarantined instead of crashing the stream. A fresh clone runs the entire seven-service stack — with health checks — from a single command, and CI verifies lint, types, tests, and the Docker build on every push.
06 · Lessons learned
Streaming guarantees are earned in failure modes, not happy paths. Designing for the restart — checkpoints, idempotence, dead letters — is what separates a demo from a pipeline.