Skip to content
PipelineZ
Get Started

Connectors

This page explains what a connector is, how pz hosts one, and how the planner decides what a connector can do. Read it before writing your own connector, or when you need to know why a read or write took a particular path.

What it is

Section titled “What it is”

A connector is the code that reads or writes one kind of place: a database, an object store, a filesystem, or an API. Every entity in connections.yml belongs to a connection, and every connection names exactly one connector, connector: postgres or connector: s3, that does the actual work of a read or a write.

Why it matters

Section titled “Why it matters”

A pipeline author never opens a driver or writes a client library call. source() and sink() name a connection and an entity, and the connector behind that connection handles the rest: authentication, pagination, retries at the request level, and turning rows into the staging database’s tables or back out again. Which connector you use, and what it declares it can do, decides what options are legal on an entity and how fast a read or write runs.

How it works

Section titled “How it works”

Builtin, external, third-party

Section titled “Builtin, external, third-party”

Fifteen connectors ship inside the pz binary and need no extra setup: localfiles, postgres, s3, sqlserver, azureblob, gcs, http, mysql, sqlite, duckdb, ducklake, motherduck, quack, iceberg, and sftp. Naming one as connector: in connections.yml is enough.

Anything else is a packaged connector: a NuGet package declared in project.yml.

project.yml
connectors:
  - package: Some.Connector.Package
    version: 1.0.0

pz restore resolves every declared package against the configured feeds, --feeds if passed, else the PZ_FEEDS environment variable, else nuget.org, downloads it under .pz/packages, and writes pz.lock.json at the project root so every machine restores the exact same version. Commit pz.lock.json, never .pz/. pz run refuses to start if pz.lock.json no longer matches project.yml’s declared connectors, unless --no-lock-check is passed.

Packaged connectors split into two tiers, the way DuckDB splits its extensions. External connectors are first-party: written, tested, and released by the PipelineZ org from pz-connector-* repos, held to the same bar as a builtin, but versioned on their own cadence and supported at a secondary tier because their dependencies don’t belong in the pz binary. External connectors lists them. Third-party connectors are anything anyone else publishes against the same ABI; pz runs them identically but PipelineZ does not test or support them.

Out-of-process execution

Section titled “Out-of-process execution”

A builtin connector runs inside the same pz process as everything else. A restored packaged connector, external or third-party, runs as its own operating-system process instead, speaking a wire protocol, PCP, back to pz over a local socket. pz spawns that process with an empty environment and repopulates only a small allowlist, PATH, HOME, and a few others: connection config and secrets never leak into the child’s environment. They cross the socket as an explicit request instead, the same way pz would configure a builtin connector in memory.

This isolation boundary is what lets a packaged connector crash, misbehave, or hold a dependency version pz itself doesn’t ship, without any of that touching the host process. It is also what a connector you write yourself has to speak, unless you’re contributing it directly into the pz binary. See Author a connector to build one, and Connector architecture for the protocol and hosting model in full.

Two data-movement tiers

Section titled “Two data-movement tiers”

For every read or write, the planner picks between two tiers: the native tier, where the connector hands DuckDB a scan or copy statement and DuckDB moves the bytes directly with no detour through pz, and the universal tier, where the connector streams Arrow record batches that pz relays into or out of the staging database. Native is faster and is what most connectors prefer; the universal tier exists for anything a native scan or copy can’t express, such as an HTTP API’s paginated response. engine.force_universal in project.yml forces every entity onto the universal tier project-wide, overriding the planner’s own choice. A connector that only implements the native tier, mysql and sqlite today, refuses a universal-only entity with PZ0312 instead of pretending to support it. A native scan that needs an unsigned, packaged DuckDB extension is also refused by default; setting allow_unsigned_extensions: true on that connection opts it back in. An entity that resumes from a sync token (a feed-shaped read, or sync: {mode: cdc}) always takes the universal tier, whatever the connector offers: the token is captured from the drained partition, and a native scan never drains one, so on the native tier the entity would replay from the same token every run. On a native-only connector that entity is refused with PZ0363.

Capabilities

Section titled “Capabilities”

A connector declares a fixed set of capabilities: flags like whether it supports a merge write, whether its commit is transactional, whether it can honor a bounded incremental window, or resume a partially delivered write. The planner reads these declarations, not the connector’s code, to decide whether an entity’s declared options are legal before any node runs, refusing a merge write on a connector with no merge capability rather than discovering that at write time. See the connector matrix for what each builtin connector declares, and Delivery guarantees for what several of those capabilities mean for a run that fails partway through.

Inspecting and testing connectors

Section titled “Inspecting and testing connectors”

pz connectors lists every connector registered to the current project, builtin and restored, with its capabilities and tiers:

Terminal window
$ pz connectors

pz connector test runs black-box protocol conformance checks against a connector package directly, independent of any project, useful while developing one:

Terminal window
$ pz connector test ./dist/my-connector --config probe.yml

Example

Section titled “Example”

Declaring a database as a read-and-write connection needs nothing beyond naming the connector and its credentials:

connections.yml
mart:
  connector: postgres
  host: ${MART_PG_HOST}
  database: mart
  user: ${MART_PG_USER}
  password: ${MART_PG_PASSWORD}
  entities:
    public.orders_current:
      write:
        strategy: merge
        keys: [order_id]

Nothing here says which tier the write uses or what merge requires: postgres declares both the Merge and NativeCopy capabilities, so the planner resolves the rest on its own.

Section titled “Related”