Lakshya

Databricks & Snowflake · Chapter 15 of 16

Governance and sharing, mechanically

How each platform actually enforces policy, what lineage does and does not capture, and the two sharing protocols that make opposite trades.

6 min read1 diagramAll 16 chapters

Both vendors make the same architectural claim — that governance executes at the query engine rather than in the application — and that claim is the right axis on which to compare them. What differs is the enforcement mechanism and, more importantly, what happens when data leaves the engine.

The namespace, and the objects that sit outside it

Unity Catalog uses a three-level namespace — catalog, schema, object — covering tables, views, volumes, functions, models and services. The detail that catches people: storage credentials, external locations, connections and shares sit directly under the metastore, not inside the three-level namespace. They are metastore-level securables, which is why grants on them behave differently from grants on a table.

The distinction between managed and external tables is about who owns the storage lifecycle, not about who governs. Both are governed identically; only one gets its files deleted when you drop it.

Lineage: what is actually captured, and the gaps that matter

Lineage is captured automatically from query execution, at both table and column level, and it spans tables, views, model versions, jobs, notebooks, pipelines and dashboards. It is genuinely useful. It is also incomplete in specific documented ways, and knowing them is what separates treating it as observability from treating it as an audit control.

Lineage does not survive an object rename. Rename a table and its history does not follow. For a system whose primary use case is impact analysis, that is a significant gap.

Column lineage fails when sources are referenced by path rather than by table name, and when a user-defined function obscures the column mapping.

Several execution paths are not captured at all — certain job submission types, RDD operations, global temporary views, and checkpointing flows.

Retention differs by surface. The catalog interface shows captured lineage indefinitely; the queryable system tables hold a rolling one-year window. If your compliance requirement is longer than a year, you must export it.

So: lineage is best-effort observability, not an audit control. Saying that, and naming the rename gap, is exactly the kind of precision that reads as experience in a governance conversation.

Row and column policies, and the performance consequence nobody mentions

Snowflake's masking and row-access policies are schema-level objects applied at query runtime. The implementation details are the interesting part:

  • A masking policy is applied at every location where the column appears — not just in the select list, but in predicates and joins too.
  • A row-access policy causes the engine to construct a dynamic secure inline view of the object, and — this is the important bit — the policy expression evaluates using the policy owner's role, not the querying role. That is what lets a policy read a mapping table the querying user cannot see.
  • Ordering is fixed: the row-access policy evaluates before the masking policy, and a column cannot appear in both a row-policy signature and a masking-policy signature.
  • Policies with simple SQL expressions perform better than ones that join a mapping table, and the documentation says so explicitly.
  • And the consequence people meet in production: aggregate queries degrade because statistics become inaccessible under a row-access policy. The metadata-only path that answers a count or a maximum without touching data no longer applies, because the engine cannot know which rows you are entitled to see without reading them. Row-level security defeats metadata-only aggregation, and that is a real and permanent cost.

Two sharing protocols that make opposite trades

DELTA SHARING — AN AUTHORISATION SERVICE THAT NEVER TOUCHES THE BYTES recipientbearer token + endpointsharing serverchecks the sharereturns capabilitypre-signed URL / temp credsclient readsdirectly from S3/ADLS/GCSprovider paysstorage AND egress Any client works — pandas, Trino, a BI tool — because the protocol is REST and the payload is Parquet. No shared platform required. The cost: a pre-signed URL or temporary credential lives outside your policy engine for its lifetime. Nothing re-checks a masking rule onthe second read. SNOWFLAKE SECURE DATA SHARING — A METADATA POINTER, NO DATA MOVES AT ALL provider creates a sharea named metadata objectconsumer mounts itread-only, zero storagequery runs in the enginepolicies apply every readconsumer pays compute onlyprovider pays storage Masking and row policies apply on every read, because the read never leaves the engine that enforces them. The cost: both parties must be on the same platform, same cloud, same region — or you pay to replicate. An open protocol that shares a capability to read files, against a closed pointer inside one engine. That is the whole comparison,and it decides which governance guarantees you keep.
An open protocol that shares a capability to read files, against a closed pointer inside one engine. The bottom line is the whole comparison, and it decides which governance guarantees survive the share.

Delta Sharing is an open REST protocol (a plain web interface over HTTP). A recipient presents a bearer token; the sharing server checks the share and returns either pre-signed URLs for individual data files or temporary cloud credentials; and the client then reads directly from object storage.

The key insight, and the thing to say in an interview: the sharing server never proxies bytes. It is an authorisation service that hands out a short-lived capability to read the provider's files. That is why any client can consume it without deploying a platform, and why the storage and egress costs sit with the provider.

Snowflake Secure Data Sharing takes the opposite approach: no data is copied or transferred. A share is a named metadata object in the provider's account; the consumer mounts it, uses no storage of their own, and pays only for the compute to query it.

The trade, stated plainly, because it decides real architectures

Delta Sharing gives you client independence — pandas, Trino, a BI tool, anything that speaks REST and Parquet — and no requirement that the recipient runs your platform. The cost is that a pre-signed URL or temporary credential lives outside your policy engine for its lifetime. Nothing re-evaluates a masking rule on the second read, because the second read does not come to you.

Secure Data Sharing keeps every read inside the engine, so masking and row policies apply every time and access history records it. The cost is that both parties must be on the same platform, in the same cloud and region, or you pay to replicate — which is a commercial constraint dressed as a technical one.

So the honest framing: choose the open protocol when the recipient's stack is not yours to choose, and accept that your governance ends at the credential. Choose the closed one when policy enforcement must survive the share, and accept the platform lock-in that buys it. Neither is strictly better, and being able to say which guarantee you are giving up is the answer.

Check yourself

What does Unity Catalog lineage not capture?Renames — lineage does not follow a renamed object. Also path-referenced sources, UDF-obscured column mappings, and several job submission types. It is observability, not an audit control.
Whose role evaluates a row access policy?The policy owner's, not the querying user's. That is how a policy can read a mapping table the user cannot see.
Why do aggregate queries slow down under row-level security?Statistics become inaccessible, so the metadata-only path that answers a count or maximum without reading data no longer applies.
What is the fundamental trade between the two sharing models?Client independence versus policy enforcement. An open protocol hands out a capability that outlives your policy engine; a closed one keeps every read inside the engine but requires both parties on the same platform.
← The cost model, from the primary documentsStreaming, and what 'exactly once' actually guarantees →