Skip to content
PipelineZ

Data Pipelines: An Article Series

A beginner-friendly article series about data pipelines: what they are, who builds them, how they are built, and what a modern data platform looks like - explained in plain language, with one running example and simple diagrams throughout. The final article introduces PipelineZ (pz) and shows how it addresses each problem the earlier articles raise.

No prior data engineering experience is assumed. If you can read a spreadsheet and have seen a SQL query before, you have everything you need.

The running example

Section titled “The running example”

Every article uses the same fictional company: Sunrise Bakery, a small chain of bakeries with an online shop. They have orders in a database, customer signups in a CRM, and payments in a third-party service - and every Monday someone asks, “how did we do last week?” Following that one question from “someone copies numbers into a spreadsheet” to “a reliable, monitored, automated pipeline” is the arc of this series.

The articles

Section titled “The articles”

Part I - Fundamentals

Section titled “Part I - Fundamentals”
  1. What is a data pipeline? - the problem pipelines solve, who builds them, and why “a script that copies data” grows into something more.
  2. What modern data infrastructure looks like - sources, lakes, warehouses, transformation, orchestration, and consumers, and how the pieces fit.
  3. Common pipeline patterns - ETL vs ELT, batch vs streaming, full refresh vs incremental, and change data capture.
  4. Data ingestion: extracting and loading - connectors, watermarks, file formats, load strategies, and delivery guarantees.
  5. Transforming data - from raw records to answers: cleaning, joining, aggregating, and layering transformations so they stay maintainable.
  6. Orchestrating pipelines - dependency graphs, scheduling, retries, parallelism, and backfills.
  7. Data validation and quality - why bad data happens, the checks that catch it, and where to put them.
  8. Monitoring and observability - knowing your pipelines ran, knowing they were right, and finding out before your users do.
  9. Best practices - the habits that separate pipelines people trust from pipelines people fear.

Part II - PipelineZ

Section titled “Part II - PipelineZ”
  1. Meet pz - a lightweight batch data pipeline engine for SQL-based ETL/ELT: its core concepts, a complete working project, how it answers each problem from Part I, and where its deliberate boundaries are.

How to read this series

Section titled “How to read this series”

Front to back if you’re new - each article builds on the previous one. If you already run pipelines for a living, Part I will read as a refresher and you can skim to Meet pz. Every diagram is an Excalidraw drawing that follows the site’s light and dark themes; click one to open it full size, and the editable .excalidraw source sits next to it under /diagrams/.