Data ingestionfor the future.

A pipeline is a file in your repo.

Whether you write it yourself or an agent drafts it, it ends up in the same place — SQL and YAML you can read, diff, and review like anything else your team ships.

pipelines/orders_enriched.sql

INSERT INTO {{ sink('lake', 'orders_curated', strategy: 'replace', format: 'parquet') }}
SELECT
    o.id,
    o.amount,
    c.email
FROM {{ ref('stg_orders') }} AS o
JOIN {{ source('raw', 'customers') }} AS c
    ON c.id = o.customer_id

connections.yml

raw:
  connector: localfiles
  entities:
    customers:
      read:
        path: data/customers.csv
        format: csv

lake:
  connector: localfiles
  root: out   # writes land under out/<entity>/

Change it, and someone reads the diff.

No config UI holds the logic hostage — a bad join or a silent strategy change gets caught in review, not in production.

pipelines/orders_enriched.sql+3 -1

-INSERT INTO {{ sink(‘lake’, ‘orders_curated’, strategy: ‘replace’, format: ‘parquet’) }}+INSERT INTO {{ sink(‘lake’, ‘orders_curated’, strategy: ‘merge’, keys: [‘id’], format: ‘parquet’) }} SELECT     o.id,     o.amount,     c.email FROM {{ ref(‘stg_orders’) }} AS o JOIN {{ source(‘raw’, ‘customers’) }} AS c     ON c.id = o.customer_id+WHERE o.amount > 0

MR

LGTM — replace was silently dropping late updates.

✓ pz validate — 2 checks passed

Ingestion, spoken into existence.

Describe the pipeline. An MCP-connected agent writes valid SQL, pz verifies every edit, and it’s running before you’ve finished your coffee.

You

Add a pipeline for total order value per customer, merged into postgres.

pz agent · via MCP

Drafting customer_order_summary.sql, validating against the current DAG…

pz mcp — customer_order_summary.sql

pipelines/customer_order_summary.sql

INSERT INTO {{ sink(‘postgres’, ‘customer_order_summary’, strategy: ‘merge’, key: [‘customer_id’]) }}
SELECT
c.id AS customer_id,
count(o.id) AS order_count,
sum(o.amount) AS total_amount
FROM {{ ref(‘stg_orders’) }} AS o
JOIN {{ source(‘raw’, ‘customers’) }} AS c
ON c.id = o.customer_id
GROUP BY c.id;

$ pz validate
validation passed (3 pipelines, 2 connections checked)
$ pz run –all
ok  customer_order_summary  3 rows 18ms

Why PipelineZ?

PipelineZ combines the power of modern data systems with a developer-friendly experience.

Blazingly Fast

Built on top of DuckDB and vectorized execution for extreme performance.

Ecosystem First

Connect to your data wherever it lives. Extensible connectors and formats.

Reliable by Design

Idempotent execution, automatic retries, and strong guarantees.

Developer Experience

YAML and SQL. Local first. Great DX from development to production.

Running in five minutes

1. Install the CLI as a .NET global tool:

   $ dotnet tool install --global pz

2. Scaffold a small, complete, runnable project:

   $ pz init demo --template sample && cd demo

   (Bare pz init scaffolds the minimal template instead — the right start for your own data. --template sample is what ships the bundled CSVs this walkthrough runs; pz init --list-templates shows all five.)

3. Run both flows — sources, pipelines, checks, and sinks in one command:

   $ pz run --all

Continue with the quickstart→

Built with
open source.

PipelineZ is 100% built with open-source projects we love. We believe in transparency, community, and open data.

View the source on GitHub→

• DuckDBAnalytics Engine
• .NETRuntime
• Apache ArrowColumnar Memory
• Apache ParquetColumnar Format
• OpenTelemetryObservability

New to data pipelines?

Data Engineering IntroductionA 10-part, beginner-friendly article series: nine vendor-neutral articles on pipeline fundamentals built around one running example, then one on how pz maps onto every one of them.