Documentation
pz is a command-line engine that moves data in batches. You describe a project as SQL files plus one YAML file of connections, and pz compiles them into a dependency graph that DuckDB executes. This page is the map of the documentation. Pick the row that matches what you need.
Start here
Section titled “Start here”| If you want to | Read |
|---|---|
| Install the tool | Install pz |
| See a project run in ten minutes | Quickstart |
| Build a real pipeline step by step | Tutorial |
| Learn the vocabulary | Key concepts |
Learn how pz works
Section titled “Learn how pz works”The concept pages explain one idea each, in the order you meet them when writing a project.
- Project layout: the files in a project and what each one does.
- Connections and entities: the one YAML declaration that describes a place and the things you read from or write to it.
- Pipelines: SQL files that read with
source()andref()and load withsink(). - Checks: assertions that run inside the graph and gate what gets written.
- Incremental loads: watermarks, bounded windows, change data capture, and merge.
- Selecting nodes: running part of a project with
--select. - How a run works: compile, plan, execute, and the artifacts a run leaves behind.
- Delivery guarantees: what pz promises when something fails halfway.
- Validation and errors: the five validation tiers and how to read an error.
- State: the
.pz/directory, watermarks, and remote state backends. - Schema contracts: column contracts, drift detection, and schema policy.
- Connectors: builtin and third-party connectors, restore, and lock files.
Do a specific task
Section titled “Do a specific task”The how-to guides are grouped by job: ingest, reliability, production, AI agents, and extending pz. Each guide states its goal, prerequisites, steps, and how to verify the result.
Look something up
Section titled “Look something up”- Connectors: one reference page per builtin connector, with every key and capability.
- CLI: every verb, flag, and exit code.
- project.yml and connections.yml: the complete key sets.
- Pipeline config, template functions, and error codes.
- Environment variables, run events, and the MCP contract.
- Versioning: what stays stable and how breaking changes are announced.
Contribute to pz
Section titled “Contribute to pz”The Internals section explains the engine for people who change it: architecture, the two-tier data plane, execution and resume internals, the connector ABI, a code tour, and the contributing guide.
Background reading
Section titled “Background reading”Data Engineering Introduction is a ten-part introduction to pipelines in general, with the last article showing how pz maps onto each idea.