Skip to content
PipelineZ
Get Started

Project layout

This page walks through the files and directories a pz project is made of, using what pz init --template sample scaffolds as the reference. Read it to orient yourself in a new project or to understand what belongs where before you start editing.

The tree

Section titled “The tree”
my-project/
├── project.yml
├── pz.lock.json          # generated by `pz restore`; commit it
├── connections.yml
├── pipelines/
│   ├── stg_orders.sql
│   ├── orders_enriched.sql
│   └── configs/
│       └── orders_enriched.yml   # optional sidecar config
└── .pz/                   # generated, gitignored
    ├── packages/           # restored connector assemblies
    ├── target/             # compiled SQL, manifest.json, plan.json
    ├── runs/<run_id>/      # run_results.json, staging.duckdb
    └── state/              # watermarks.json, sync-state.json

pz init writes project.yml, connections.yml, and a .gitignore that excludes .pz/. The pipelines/ directory and its contents exist only once you add a first pipeline, unless you scaffold from a template that ships one.

project.yml

Section titled “project.yml”

This is the project’s identity: its name, version, the connector packages it needs, project variables, and engine settings. A minimal file needs only name and version:

project.yml
name: pz_new_project
version: 0.1.0


retention:
  keep_last: 10

Builtin connectors need no declaration. A connector shipped as a separate NuGet package is listed under connectors: with a package and version, and pz restore resolves it. Other keys cover project vars:, engine: settings such as thread count and DuckDB memory limits, retention: for automatic disk cleanup, and state: to move run state off the local disk. The full key reference lives at project.yml reference.

pz.lock.json

Section titled “pz.lock.json”

pz restore writes this file at the project root after resolving every declared connector package to an exact version. It is generated, but it is not gitignored: commit it so every machine and CI run resolves the identical package graph. pz run and pz validate check it against project.yml and refuse to proceed on drift unless you pass --no-lock-check.

The lock pins content as well as versions: every installed file carries a sha512. A later pz restore restores exactly the locked versions and refuses a package whose download hashes differently (PZ0327), and loading refuses an installed file that changed after restore (PZ0326). To move to newer versions after editing project.yml, run pz restore --update, which re-resolves every connector and rewrites the lock.

connections.yml

Section titled “connections.yml”

One file, at the project root, that declares every place the project talks to. Each top-level key is a connection: a connector: name plus that connector’s own config, and an entities: map naming what you read or write there.

connections.yml
raw:
  connector: localfiles
  entities:
    customers:
      read:
        path: data/customers.csv
        format: csv
        columns:
          id: bigint
          email: varchar


lake:
  connector: localfiles
  root: out

The full shape, including the reserved connection keys and the read/write option surfaces, is covered in Connections and entities.

pipelines/*.sql

Section titled “pipelines/*.sql”

Every .sql file under pipelines/ is a pipeline, named after its file. There is no registration list: dropping a file into the folder is what makes it part of the project.

pipelines/orders_enriched.sql
INSERT INTO {{ sink('lake', 'orders_curated', strategy: 'replace', format: 'parquet') }}
SELECT
    o.id,
    o.amount,
    c.email
FROM {{ ref('stg_orders') }} AS o
JOIN {{ source('raw', 'customers') }} AS c
  ON c.id = o.customer_id

See Pipelines for how source(), ref(), and sink() build the dependency graph from this file alone.

pipelines/configs/<name>.yml

Section titled “pipelines/configs/<name>.yml”

An optional sidecar, matched to its pipeline by name, that declares materialization, tags, and data-quality checks. It is optional: a pipeline with no sidecar defaults to materialization: table and no checks.

pipelines/configs/orders_enriched.yml
pipeline: orders_enriched
materialization: table
tags: [daily]
checks:
  - not_null: [id, email]
  - unique: [id]

See Checks for the check types, and the full sidecar key reference at Pipeline config reference.

.pz/

Section titled “.pz/”

Everything pz generates lives under .pz/, gitignored by default. pz init’s .gitignore excludes the whole directory.

  • .pz/packages/ holds restored connector assemblies, materialized by pz restore.
  • .pz/target/ holds compiled artifacts: rendered SQL, manifest.json (the full node and edge graph), and plan.json (how each edge will physically move data). pz compile and pz plan write here.
  • .pz/runs/<run_id>/ holds one directory per run: run_results.json and, until swept by retention, that run’s staging.duckdb. See How a run works.
  • .pz/state/ holds watermarks.json for incremental cursors and sync-state.json for connector-managed continuation tokens. See Incremental loads.

pz clean reclaims disk from .pz/runs without touching .pz/state, .pz/target, or .pz/packages.