Skip to content
PipelineZ
Get Started

Inspect and validate

This guide shows you how to check a project before running it: preview the execution plan, validate configuration and SQL, and compile without executing. All three verbs are read-only; no data moves. It also covers fixing a watermark that has advanced too far.

Prerequisites

Section titled “Prerequisites”
  • A runnable project. Follow the quickstart to scaffold one.

Steps

Section titled “Steps”

  1. Preview the execution plan with pz plan, to see which nodes exist and how each will execute:

    Terminal window
    $ pz plan
    strategy      node                     reason
    native_scan   src_raw__customers       native scan: connector 'localfiles' provides read_csv over data/customers.csv
    native_scan   src_raw__orders          native scan: connector 'localfiles' provides read_csv over data/orders.csv
    duck_sql      stg_orders               duckdb sql: executes in-engine
    duck_sql      order_totals             duckdb sql: executes in-engine
    duck_sql      orders_enriched          duckdb sql: executes in-engine
    native_copy   lake.order_totals        native copy: connector 'localfiles' provides COPY TO csv
    native_copy   lake.orders_curated      native copy: connector 'localfiles' provides COPY TO parquet
    memory budget: ~1.63 GB (duckdb 1.00 GB + channels 0.38 GB + overhead 256MB)

    No nodes execute. The output shows each node’s execution strategy, native scan or copy where the connector can hand work straight to DuckDB, universal-path duck_sql otherwise, and the static memory budget for the run.

  2. Validate configuration and SQL with pz validate, to catch errors before a run:

    Terminal window
    $ pz validate
    validation passed (3 pipelines, 2 connections checked)

    This runs validation tiers 1 to 4: project load, compilation, configuration checks, and a SQL dry-compile against DuckDB.

  3. Compile only, with pz compile, to render pipelines and build the DAG without executing or planning:

    Terminal window
    $ pz compile

    This is the fastest way to check that templating resolves and the DAG is well-formed, for example in a pre-commit hook.

Verify

Section titled “Verify”

pz plan, pz validate, and pz compile all exit 0 on success. A clean pz validate run prints validation passed with the pipeline and connection counts; anything else is a configuration problem to fix before you run the project for real.

Fix a wrong watermark

Section titled “Fix a wrong watermark”

A watermark that has advanced past data you still need means the next run skips it. Fix it with pz state, never by editing .pz/state/watermarks.json directly: the verb validates the value, refuses while a run is in flight, and records what it replaced.

  1. See what is stored, and what past runs recorded:

    Terminal window
    $ pz state show erp.dbo.orders

    The history section lists every run that recorded a watermark for this entity, newest first, with that run’s own status. This is the menu you pick a rollback target from.

  2. Roll back to the position before the bad run. Each run’s recorded value is what that run advanced to, so rolling back to run N-1 restores the state as of before run N:

    Terminal window
    $ pz state rollback erp.dbo.orders --to-run 20260727T020009111Z-3f2e --reason "late-arriving rows"

    It prints the before and after values and what the next run will re-extract, then asks for confirmation. Add --dry-run to preview without writing, or --yes to skip the prompt.

  3. If no run has the value you need, set it directly. The cursor column and type come from the stored entry, so only the value is yours to get right:

    Terminal window
    $ pz state set erp.dbo.orders --value 2026-07-01 --reason "skipping corrupt batch"

  4. If the entry itself is broken, a cursor type pz doesn’t recognize, remove it and let the next run extract in full:

    Terminal window
    $ pz state clear erp.dbo.orders --reason "corrupt entry"

Every write appends one line to .pz/state/audit.jsonl recording the replaced value, the new one, and your --reason. pz clean never touches that file.

Troubleshooting

Section titled “Troubleshooting”
If you see Do
pz validate fails with a PZ#### error Fix per the error’s own file, line, and message. See Error codes.
A watermark has advanced past data you still need Roll it back with pz state rollback. See Fix a wrong watermark above.
pz state show exits 1 The state file is corrupt. Restore it from a backup, or delete it and accept that the next run extracts everything in full.
pz plan shows duck_sql where you expected a native scan or copy The connector has no native path for that format, or engine.force_universal is set project-wide in project.yml.
Section titled “Related”