Diagrams

Five diagrams walk from overview to detail, explaining how pz works. Every code, YAML, or SQL snippet in them is real content from templates/sample. This page is the gallery; each section links to the page that explains its content in prose.

Diagram 01: overview

Overview: a pz project of YAML and SQL compiles to a DAG that DuckDB executes, producing data and a run receipt — Click the diagram to open it full size.

The whole machine in one picture: a project of YAML and SQL compiles into a DAG, DuckDB executes it as the hub, and the run produces both data and a machine-readable receipt. It also lays out the eight phases every verb shares. Explained in Architecture.

Diagram 02: compiling to a DAG

Compile: ref(), source() and sink() calls observed during rendering declare the DAG edges — Click the diagram to open it full size.

Zooms into compilation: how ref(), source(), and sink() calls observed during template rendering declare the DAG’s edges, with no SQL parsing involved, using templates/sample verbatim. Explained in Execution internals.

Diagram 03: the data plane

The data plane: native scan versus streamed Arrow batches, chosen per edge — Click the diagram to open it full size.

Zooms into how bytes physically move: the two-tier data plane, native scan/copy versus the universal Arrow batch stream with bounded channels and stall-based bottleneck diagnostics, plus how much data a single run claims (full load, incremental, or a bounded window). Explained in The data plane.

Diagram 04: run lifecycle

Run lifecycle: topological dispatch, one typed event stream, and retry semantics — Click the diagram to open it full size.

Zooms into pz run actually executing: the topological dispatcher under engine.threads, the single typed event stream that feeds both the console tree and NDJSON, and failure/retry semantics down to exit codes. Explained in Execution internals.

Diagram 05: resilience and resume

Resilience: four containment tiers, the progress records that survive a failure, and the delivery-guarantee matrix — Click the diagram to open it full size.

The failure story end to end: four containment tiers ordered by blast radius, the progress records that a failure cannot destroy (watermark, sync state, partition and delivery ledgers), and the delivery-guarantee matrix those records exist to uphold. Explained in Resume internals.

Architecture: the design principles and layering behind diagram 01.
Execution internals: the compiler and dispatcher behind diagrams 02 and 04.
The data plane: the two tiers behind diagram 03.
Resume internals: the ledgers behind diagram 05.