"Spark. Postgres. Duplicates. Dang it." — Daniel Beach (Data Engineering Central)
Beach walks through an intermittent-duplicate-records debugging story across a Delta Lake → Databricks Spark → Python → Postgres → Web App pipeline. The article is a fundamentals re-up rather than a new technique. The body the email exposes stops before the actual root-cause reveal (Substack truncates mid-investigation at the Delta Lake table step).
Two reusable mental models he names explicitly:
- "What is the grain of this dataset?" Before you call something a duplicate, define what makes each record unique. Duplicates are sometimes by design (faulty assumption, bad decision, but intentional). Grain question is the first move, not the bug hunt.
- Source / Transformation / Destination decomposition. When duplicates surface in a complex pipeline, partition the search space into three boxes before chasing the bug. Reduces the surface area you have to reason about. Eliminate from the systems you control before blaming systems you don't.
Closing principle: "Control the controllables." Eliminate duplicates in your own systems first; don't try to debug the whole pipeline simultaneously.
⚠️ Sponsorship
Sponsored by Delta (Delta Lake / Databricks) — top-of-article thank-you sponsor block, no inline ad unit. Author Daniel Beach disclosed the relationship in the standard DEC sponsor slot. The article content (Spark + Postgres duplicate-row debugging story) is technically independent of the sponsor's product, but readers should note that DEC's curation/topic-selection may favor data-engineering surfaces where Databricks tooling is the natural answer.
Why this is in the vault
Two reasons to file despite the weak RDCO surface mapping:
- The grain question and the source/transformation/destination decomposition are pipeline-agnostic. They apply directly to Ray's own data-ingestion pipelines (Gmail newsletter intake → vault note → graph re-ingest; iMessage/Discord intake → memory; web research → vault). When a vault note appears twice or a graph edge double-fires, the same first move applies: what's the grain of this artifact, and which of the three boxes (source intake, transformation, destination write) owns the dup.
- Continues the Daniel Beach / DEC cadence tracking — fourth fundamentals-flavored DEC piece in ~6 weeks ([[2026-04-17-data-engineering-central-architectural-principles]], [[2026-04-22-data-engineering-central-most-teams-doing-it-wrong]], [[2026-04-29-data-engineering-central-ai-changing-de-fast]], this one). The pattern is consistent: Beach keeps re-anchoring his audience on fundamentals while the discourse chases AI tooling. That's a positioning signal worth tracking for Sanity Check's adjacent audience.
No Sanity Check angle here. The duplicate-debugging story is too well-trodden to re-frame originally, and there's no novel claim to push back against. File for cross-link surface, not for content fodder.
Mapping against Ray Data Co
Weak mapping. The most defensible connection points:
- MAC framework ([[../01-projects/data-quality-framework/testing-matrix-template.md]]). The grain question maps to MAC's row-uniqueness basis cell. Beach's article is implicit evidence that even experienced engineers skip the grain definition step and jump to "find the bug" — which is exactly what MAC formalizes as a required acceptance test rather than an ad-hoc detective exercise. If the body had survived the truncation, this would be a candidate citation for the MAC anchor article's "why fundamentals matter" intro.
- Ray's own ingestion pipelines. The /process-newsletter pipeline (Gmail → subagent extraction → vault note write) has a source/transformation/destination shape and could generate duplicates at any of three points: Gmail thread re-delivery, subagent re-running on the same message_id, or vault note re-write on filename collision. The current dedup story is filename-based at the destination only. Beach's decomposition is a reminder to check the source (Gmail message_id idempotency) and transformation (subagent dispatch idempotency) layers too, not just the write step.
- Sponsor flag: Delta Lake (Databricks). Beach has disclosed the relationship at the top of the article. Not a hidden ad. Worth noting because the article's pipeline-as-described starts at "Delta Lake House" — the sponsor relationship is at least partially shaping which tech stack he picks for the demo. Doesn't invalidate the fundamentals lesson but flags that "everything starts at Delta Lake" is a sponsorship-shaped framing, not a universal default.
Action: no follow-up. Cross-link only. If the full body becomes accessible and reveals an unusual root cause (e.g., a Spark write-mode subtlety or a Postgres UPSERT race), revisit and upgrade.
Related
- [[2026-05-11-dataengineeringcentral-pyspark-testing-runtimes.md]] — same author, adjacent fundamentals-vs-AI-tooling posture
- [[2026-04-29-data-engineering-central-ai-changing-de-fast.md]] — Beach's framing rant on AI adoption styles; consistent with his "control the controllables" stance here
- [[2026-04-22-data-engineering-central-most-teams-doing-it-wrong.md]] — Beach interviewing Chris Gambill; same "teams skip fundamentals" thesis
- [[2026-04-17-data-engineering-central-architectural-principles.md]] — Beach's Part 3 on architectural foundations
- [[../01-projects/data-quality-framework/testing-matrix-template.md]] — MAC Scope × Basis; grain question lives here
- [[2026-03-17-seattle-data-guy-full-refresh-vs-incremental.md]] — adjacent pipeline-shape debugging