Skip to content
PipelineZ
Get Started

Connectors

A connector is the code that reads or writes one kind of place: a database, an object store, a filesystem, or an API. pz ships fifteen builtin connectors, and every one of them can act as a read connection, a write connection, or both in the same project.

Capability matrix

Section titled “Capability matrix”

“Native DuckDB tier” means the connector can hand DuckDB a native scan or copy instead of streaming rows through Arrow. “Incremental” and “CDC” name the capability flags that gate a windowed read or a change-capture read. “Merge” names keyed upsert writes.

Connector Read Write Native DuckDB tier Incremental CDC Merge Formats
localfiles ✓ ✓ ✓ ✓ – – csv, tsv, parquet, json, xlsx, avro (read)
postgres ✓ ✓ – ✓ ✓ ✓ –
sqlserver ✓ ✓ – ✓ ✓ ✓ –
mysql ✓ ✓ ✓ ✓ – – –
sqlite ✓ ✓ ✓ ✓ – – –
duckdb ✓ ✓ ✓ ✓ – ✓ –
ducklake ✓ ✓ ✓ ✓ – ✓ –
motherduck ✓ ✓ ✓ ✓ – ✓ –
quack ✓ ✓ ✓ ✓ – ✓ –
iceberg ✓ ✓ ✓ ✓ – ✓ –
s3 ✓ ✓ ✓ ✓ – – csv, tsv, parquet, json, xlsx (read), avro (read)
azureblob ✓ ✓ ✓ ✓ – – csv, tsv, parquet, json, xlsx (read), avro (read)
gcs ✓ ✓ ✓ ✓ – – csv, tsv, parquet, json, xlsx (read), avro (read)
sftp ✓ ✓ – ✓ – – csv, tsv, parquet, json
http ✓ ✓ – ✓ – ✓ json

Database connectors have no file format: postgres, sqlserver, mysql, sqlite, duckdb, ducklake, motherduck, quack, and iceberg move rows, not files. mysql, sqlite, the four DuckDB-family connectors (duckdb, ducklake, motherduck, quack), and iceberg run on the native tier only, so declaring engine.force_universal for one of their entities fails with PZ0312. quack’s merge is a whole-table rewrite rather than a keyed upsert; its page explains what that costs.

Packaged connectors

Section titled “Packaged connectors”

A connector that does not ship with pz is a NuGet package, declared in project.yml under connectors:. That covers two tiers: the external connectors the PipelineZ org itself publishes outside the binary (first-party code at a secondary support tier, listed on External connectors), and third-party connectors anyone can publish. Both are declared and restored the same way:

project.yml
connectors:
  - package: Some.Connector.Package
    version: 1.0.0

pz restore resolves every declared package against the configured feeds (--feeds, else PZ_FEEDS, else nuget.org), downloads it under .pz/packages, and writes pz.lock.json so every machine restores the exact same version. A restored connector runs out-of-process, the same isolation boundary a builtin connector runs inside. See Connectors: the plugin architecture for how that isolation works, and Author a connector to write your own.

Test a connection

Section titled “Test a connection”

pz validate --connect is how you check that a connection in connections.yml is actually reachable. It runs the full config and SQL validation, then probes live connectivity and schema drift for every declared entity:

Terminal window
pz validate --connect

If you are building a connector instead, pz connector test <target> runs black-box protocol conformance checks against its package directory or entrypoint binary, using a --config file that names the connection and the read:/write: entities to probe:

Terminal window
pz connector test ./dist/my-connector --config test-config.yml
Section titled “Related”