← Back to work

Real-time data · 2026

Streaming Clickstream Pipeline

A real-time lakehouse for e-commerce clickstream — Kafka to Spark to Delta Lake to a live dashboard. Seven services, one command, exactly-once semantics.

events ingested
~50/s
micro-batch cadence
10 s
services, one command
7

Architecture

Select a stage to explore it.

01 · Event generator

Synthesizes ~50 clickstream events per second — page views, searches, cart actions, purchases — including deliberately invalid messages to exercise the dead-letter path.

Docker Compose — 7 services with health checksGitHub Actions — lint · types · tests · builddead-letter queue for invalid events

Visuals

Streaming Clickstream Pipeline GitHub repository with architecture documentation
Architecture documented in the repo — Kafka, Spark, Delta Lake, DuckDB, Streamlit

01 · Problem

Batch pipelines answer yesterday's questions. E-commerce platforms emit a continuous stream of behavioral events, and understanding conversion funnels or product performance in real time means ingesting, processing, and serving those events with minimal latency — without losing any when things restart.

02 · Solution

An event-driven pipeline that moves a click from the browser to an analytics dashboard in under a minute: Kafka for durable ingestion, Spark Structured Streaming for processing, a Delta Lake medallion architecture on S3-compatible storage, DuckDB for serving, and Streamlit for the live view. The whole stack starts with one command.

03 · Engineering decisions

Delta Lake over raw Parquet: ACID transactions and schema evolution are worth the metadata overhead. DuckDB over a warehouse: zero-config and fast, accepting no multi-user concurrency — right-sized for the workload. One foreachBatch query for all three layers: simpler checkpoint management, at the cost of a shared processing cadence. MinIO over cloud S3: fully reproducible local development, one config change away from AWS.

04 · Challenges & trade-offs

The hard part of streaming is failure, not throughput. The pipeline achieves effectively-exactly-once semantics through idempotent Delta writes plus Kafka offset tracking, recovers from restarts via checkpointing and consumer-group rebalancing, and handles out-of-order events with watermarking — while a 7-day range filter drops stale replays.

05 · Results

Sub-minute latency from raw event to dashboard, with invalid events quarantined instead of crashing the stream. A fresh clone runs the entire seven-service stack — with health checks — from a single command, and CI verifies lint, types, tests, and the Docker build on every push.

06 · Lessons learned

Streaming guarantees are earned in failure modes, not happy paths. Designing for the restart — checkpoints, idempotence, dead letters — is what separates a demo from a pipeline.