06-reference

data engineering weekly privacy safe data platform

2026-07-10·reference·source: Data Engineering Weekly·by Ananth Packkildurai
data-privacyprivacy-engineeringtokenizationpseudonymizationPII-classificationdata-governanceenterprise-data-platformAI-guardrailscompliancedata-engineering

"Beyond Redaction: Anatomy of a Privacy-Safe Data Platform" — @ananthdp

Why this is in the vault

This is one of Ananth's most architecturally dense standalone essays. It lays out a five-stage privacy pipeline (Discover → Authorize → Protect → Use → Prove) with enough implementation specificity to drive both client evaluation conversations and internal platform design. The six demo scenarios in the evaluation section are directly deployable as a phData discovery checklist. The LLM/RAG boundary analysis maps to RDCO's agentic Cortex AI architecture decisions.

The core argument

The hard problem in privacy-safe data platforms is not making personal data disappear — it is preserving the minimum useful properties for an allowed purpose, limiting who can link the data, and producing evidence that the controls actually ran.

The guiding question shifts from "Can it hide this column?" to "What property must this recipient retain, for which permitted purpose, and what evidence will prove enforcement?"

The five-stage pipeline

1. Discover and classify candidates — not certainty

Cheap signals over expensive ones: column-name heuristics → pattern recognizers → NLP/contextual analysis. Output should be candidates with data type, sensitivity tier, confidence, evidence, and recognizer version — not a boolean is_pii. Classification drifts; treat it as versioned and recurring, not a one-time migration.

2. Authorize the use, not merely the field

Policy evaluation must consider: data class, principal/role/tenant, declared purpose, recipient, jurisdiction/contract/legal basis, consent/opt-out state, required control, and approval path. Consent is one input, not a universal switch (cites India DPDP Act and HIPAA as examples where it is insufficient). Practical pattern: policy-as-data with versioning, approval, simulation, and rollback. Lifecycle: propose → simulate → approve → enforce → verify → rollback. Enforce at point of use, not as an overly optimistic proxy.

3. Select the smallest control that preserves the required utility

Three modes, three different promises:

Key clarifications on common confusions:

4. Safe exchange is a controlled release, not a file write

Steps: authorize recipient/purpose/destination/retention → transform only approved fields → re-scan with versioned detectors → quality check (referential integrity, token-domain consistency, row/column counts) → produce signed manifest (hash, transformation version, policy version, detector config, findings summary, approver, expiry) → deliver through revocable access channel → record receipt/revocation. The manifest says what it actually proves (artifact integrity and provenance), NOT that data contains no PII.

5. Evidence first — a hash chain is a building block, not a guarantee

Lineage chain: policy version → request → decision → transformation → enforcement result → delivered artifact. Hash-chained logs detect changes but don't defend against an attacker rewriting the DB and recomputing hashes. Credible design adds: write-once or independently held storage, signatures with externally verifiable checkpoints, separation between audited service and log store, periodic review. Cites NIST SP 800-53 AU-9.

The AI/LLM boundary

Controls must apply before, during, and after generation. Enforce tenant/purpose filters outside the model; treat tool results and model output as untrusted. MCP tool calls (e.g., classify_text) are not enforcement boundaries — prompt injection can bypass them. Policy enforcement must sit outside the model. Cites OWASP Sensitive Information Disclosure.

The TL;DR control map

Use case Control
Internal approved tables Purpose-bound deterministic tokenization
External partner needs insight Aggregation, protected query, or clean-room workflow
Authorized staff need to re-identify Vault-backed reversible tokenization
Auditor needs proof Signed manifest + protected audit trail
LLM/RAG workflow Retrieval authorization + output guardrails

Six evaluation scenarios (the demo checklist)

  1. Utility — join on stable pseudonym without direct identifiers
  2. Bounded linkage — different tokens per recipient; joins only within domain
  3. Governed recovery — demonstrate denial and approval with rate limit and evidence
  4. Release assurance — show policy decision, transformation version, detector result, signed manifest, expiry, revocation
  5. Tenant isolation — attempt cross-tenant reads/tokenization/evidence lookup
  6. AI resistance — show unauthorized documents can't enter retrieval; prompt-injected tool output can't exfiltrate

Mapping against Ray Data Co

The six evaluation scenarios above are the most immediately deployable asset here — Ben is running discovery and scoping for phData enterprise clients who are standing up or modernizing data platforms. Regulated-industry clients (financial services, healthcare, security companies like KnowBe4) routinely face exactly these architecture decisions. Walking into a discovery call with this checklist converts vendor conversations from marketing pitches into evidence-based assessments.

The policy-as-data lifecycle (propose → simulate → approve → enforce → verify → rollback) maps directly to the Fabric governed knowledge graph framing. Fabric's value proposition is that governance is not a layer bolted onto pipelines — it is the primitive. This essay provides the implementation-level vocabulary for that claim.

The per-recipient token isolation concept (prevent cross-recipient correlation) directly extends the multi-tenant isolation problem in RDCO's agentic Cortex AI architecture. The principle "tenant isolation must be in the design, not just a tenant_id column" is a clean heuristic for CAF client delivery.

The AI/LLM section's hardest insight is also the most useful for Cortex AI engagements: enforcement must live outside the model. MCP tool boundaries and prompt patterns are not enforcement surfaces. This reframes where phData builds controls in agentic architectures.

FF3-1 deprecation is a concrete, citable fact for any client currently running FPE tokenization — worth flagging in technical assessments.

Related