PipelineZ documentation

PipelineZ (pz) is a lightweight, developer-first batch data pipeline engine for SQL-based ETL/ELT, powered by DuckDB, that can run anywhere without requiring a data platform. Start with the quickstart, then pick the page type you need: how-to guides for tasks, concepts for understanding, reference for exact contracts.

Get started

• Quickstart: run your first pipeline
• Key concepts — DAG, node, source, sink, and the rest of the vocabulary
• A code tour for new contributors — follows one pz run from the keystroke to the output files, naming the actual classes at every stop, in plain terms

How-to guides

• Inspect and validate a project
• Handle schema drift
• Detect schema drift at run time
• Run checks and retry failures
• Secure connection config
• Observe runs with Azure Monitor
• Run scheduled on Windows
• Move state off the local disk
• Tune retries per database
• Backfill in bounded slices
• Throttle a struggling source or sink
• Extract from an HTTP API
• Use Google Cloud Storage
• Capture changes with CDC
• Author a connector
• Use pz with an AI agent

Concepts

• Architecture overview
• Project structure
• The data plane
• The execution model
• Connectors
• Delivery guarantees
• Validation and errors
• Contributor internals

Reference

• CLI verbs and exit codes
• project.yml reference
• MCP contract reference
• Run events (NDJSON contract)
• Versioning and breaking changes
• Performance and memory
• Diagram set