Databricks Lakeflow

Serve BARRY data to Databricks on demand.

The Lakeflow Connector lets Databricks pull data from BARRY into Unity Catalog using Databricks' Lakeflow Community Connectors framework. You build a Transport whose destination is a Lakeflow connection; a small Python connector running inside your Databricks workspace then calls BARRY on demand and lands the rows in a Unity Catalog table.

Unlike every other destination, a Lakeflow transport stores nothing — it is a pull endpoint. When the Databricks pipeline reads the table, BARRY reads the on-prem source live and streams the rows straight back as Apache Arrow.

• Pull-based, direct query. Every pipeline run is a fresh query — there is no copy, no schedule, and no run button on the BARRY side.
• Snapshot only. This connector does direct queries, not change-data-capture.
• One token per group → many tables. A Lakeflow access token is scoped to a transport group: list_tables() returns every Lakeflow-sink transport in the group, and one Databricks pipeline ingests them all. Each transport in the group is one served table.
• Cloud-served, on-prem source. Serving always terminates at the cloud server; each transport's source must be a client-placed (on-prem) connection because the server cannot reach an on-prem source itself for Databricks to pull.

How it works

1. Databricks calls the BARRY serving API for the table.
2. BARRY queues a one-shot run; the on-prem client (whose heartbeat is already open) picks it up immediately.
3. The client runs the query and pushes Apache Arrow batches to the server.
4. The server streams those batches straight back to Databricks — no staging.

Each pull is a normal run, so it appears on the Monitor page (Runs tab) with rows, bytes, duration and throughput, exactly like any other connector.

Set up in BARRY

1. Create a Lakeflow connection — New connection → Lakeflow. There is nothing to configure beyond a name (the connection holds no credentials). It is marked Experimental, so enable experimental features in Settings if you don't see it in the picker.
2. Create a group for the tables you want Databricks to ingest together (Transports → Manage groups), or pick one in the wizard.
3. Create one transport per table — pick an on-prem source (e.g. SQL Server) and the table to expose, choose the Lakeflow connection as the destination, and assign the group. Optionally set a served table name (defaults to the source object name). No Output Format is needed. A Lakeflow transport must belong to a group.
4. Issue the group's access token — in Manage groups, open the group's Lakeflow access card → Issue token. Copy the values shown — the token is shown only once:
   • Server URL — your BARRY base URL.
   • Token — a per-group, revocable access token (serves all the group's tables).
   • Tables — the names Databricks will see.

A Lakeflow transport cannot be run manually or on a schedule and takes no triggers — Databricks starts it by pulling. The token is generated per group (one token serves every Lakeflow table in the group); re-issuing rotates it and revoking it from the same card stops access immediately.

Set up in Databricks

Prerequisites: a Unity Catalog–enabled workspace, workspace-admin approval to use community connectors (the framework is Beta), Python 3.10+, and a target catalog + schema.

1. Deploy the BARRY connector into your workspace following the Databricks community-connector flow (the connector ships in the lakeflow-connector/ folder of the BARRY distribution, with its own README).
2. Create a Unity Catalog connection — as Auth Type choose USES_ANY_STATIC_CREDENTIAL (the BARRY token is a static bearer token; do not pick USES_OAUTH_M2M — BARRY exposes no OAuth token endpoint). Enter the parameters from the group's BARRY Lakeflow card with exactly these key names, either in the named fields or as Additional-Options rows:
   • base_url — the Server URL.
   • token — the access token (sent as a Bearer token; stored securely in UC).
   • verify_ssl — optional, defaults to true; false only for dev servers with self-signed certificates.
3. Create and run an ingestion pipeline using the BARRY community connector: select the connection, pick the tables (the connector lists every Lakeflow table in the group), choose the target catalog + schema, and run.

Use it

• Refresh by re-running the Databricks pipeline (manually or on its own schedule). Each run does a fresh snapshot query against the live source.
• Verify in Databricks by querying the target table, and in BARRY by opening the transport's Runs — each pull shows there with full metrics.
• Rotate or revoke the token from the group's Lakeflow access card (Manage groups); update the Unity Catalog connection after a rotation.

What to expect

• Built on the Databricks community-connector framework, which Databricks currently ships as Beta.
• On-demand source reads. Serving terminates cloud-side and the on-prem source is queried live, so the first rows arrive within a few seconds.
• One table per transport, one token per group. A group's access token serves every table in it, and each read delivers a complete, current snapshot.