BARRYBARRY
Documentation

PySpark connector

Read BARRY data from Spark.

The PySpark Connector lets a Databricks or Spark 4 notebook read on-prem data straight into a Spark DataFrame from BARRY — no materialized destination and no format conversion. It registers a Spark DataSource named barry; when the notebook reads a table, BARRY reads the on-prem source live and streams the rows back as Apache Arrow, which PySpark loads natively.

It shares the serving path with the Lakeflow Connector, but is decoupled from the Lakeflow destination: you expose a table by giving a transport the Spark Direct Barry destination (a virtualized, storage-less sink) and putting it in a transport group — no Lakeflow sink needed. The whole surface is gated by an admin master switch (Settings → Serving (PySpark)), off by default.

  • Pull-based, direct query. Every read is a fresh query — no copy, no schedule.
  • Snapshot only. Direct queries, not change-data-capture; no deletes.
  • One token per group → many tables. The access token is scoped to a transport group and is the same token the Lakeflow connector uses. spark.read sees every servable table in the group.
  • Cloud-served, on-prem source. Serving terminates at the cloud server; the transport's source must be client-placed because the server cannot reach an on-prem source itself.

How it works

  1. The notebook reads spark.read.format("barry")… for a table.
  2. BARRY queues a one-shot run; the on-prem client (heartbeat already open) picks it up immediately.
  3. The client runs the query and pushes Apache Arrow batches to the server.
  4. The server streams those batches straight back to Spark — no staging.

Each read is a normal run, so it appears in Monitor with rows, bytes, duration and throughput, exactly like any other connector.

Set up in BARRY

  1. Enable serving — an admin turns on Settings → Serving (PySpark) (off by default; while off the serving API is invisible).
  2. Create a group for the tables you want to expose (Transports → Manage groups), or pick one in the wizard.
  3. Add transports with the Spark Direct Barry destination — each with an on-prem source (e.g. SQL Server), the table to expose, and the Spark Direct Barry connection as the destination. No Output Format is needed. Optionally set a served table name (defaults to the source object name).
  4. Issue the group's access token — in Manage groups, open the group's Serving access card → Issue token. Copy the Server URL and token (the token is shown only once). The same token authorizes both the Lakeflow connector and PySpark; scope the group to what you intend to expose.
Re-issuing rotates the token; revoking it from the same card stops access immediately.

Set up in the notebook

Prerequisites: Databricks Runtime 15.4 LTS+ (or serverless environment v2), or Apache Spark 4.0+ — the Python Data Source API is required. Store the token in a Databricks secret scope; don't inline it.

1. Install the connector — download the wheel with the ↓ Barry Wheel button on the Transports page (always the latest), then install it (full step-by-step: Using the Barry wheel):

%pip install /Volumes/<catalog>/<schema>/<volume>/barry_io_pyspark-<version>-py3-none-any.whl
dbutils.library.restartPython()

2. Register and read:

from barry_io_pyspark import BarrySparkDataSource
spark.dataSource.register(BarrySparkDataSource)

df = (spark.read.format("barry")
        .option("base_url", "https://barry.example.com")
        .option("token", dbutils.secrets.get("barry", "serve-token"))
        .option("table", "Customers")
        .load())
df.show()

Options: base_url, token, table (all required), and verify_ssl (default true; false only for dev servers with self-signed certificates).

On Databricks the runtime already ships PySpark, so the wheel does not pull its own. Off-cluster (local Spark), install the Spark extra: pip install "barry-io-pyspark[spark]".

Use it in a plain Python notebook (no Spark)

The same wheel works without Spark. A light install — just requests and pyarrow, no PySpark — gives you the served table as Apache Arrow, one call:

%pip install /Volumes/<catalog>/<schema>/<volume>/barry_io_pyspark-<version>-py3-none-any.whl

from barry_io_pyspark import read_table
tbl = read_table("https://barry.example.com", token, "Customers")  # -> pyarrow.Table
df  = tbl.to_pandas()                                              # or Polars / DuckDB

read_pandas(base_url, token, table) returns a pandas DataFrame directly (needs pandas). For more control — listing tables, the schema, or streaming batches — use the client:

from barry_io_pyspark import BarryClient
client = BarryClient("https://barry.example.com", token)
client.list_tables()                       # servable tables in the token's group
client.get_schema("Customers")             # column names, types, nullability
for batch in client.read_arrow_batches("Customers"):
    ...                                    # incremental Arrow RecordBatches

Same serving API, same per-group token, same complete-snapshot semantics as the Spark path — it simply lands in Arrow/pandas instead of a Spark DataFrame.

What to expect

  • A complete snapshot on every read. Each spark.read runs a fresh query and delivers the table as it stands right now — no staging copy to keep in sync.
  • Parallelism scales across tables. A table is read as one partition; reading several tables runs as several Spark jobs, in parallel.
  • You read the transport's configured shape. The table arrives as the transport defines it; a Spark .filter(...) applies to the DataFrame once the snapshot is in.
  • On-demand source reads. The on-prem source is queried live, so the first rows arrive within a few seconds.
Previous
← VS Code extension
Next
Python wheel →