DEDP 6.1 — Data Engineering Design Patterns (Intro)

This chapter draws the line between Data Engineering Patterns (DEPs) and Data Engineering Design Patterns (DEDPs). DEPs are immediately implementable building blocks — caching, reusability, workspace packaging. DEDPs are higher-level architectural blueprints: they tell you how to compose those building blocks into systems that solve recurring strategic problems.

The Core Definition

A DEDP is a "general repeatable solution to a commonly occurring problem" — borrowed from software design pattern vocabulary, applied to data engineering's specific challenges. These are not tool recommendations. They are structural decisions about data flow, modeling, governance, and platform integration.

What DEDPs Address

Five strategic concerns keep appearing across data orgs:

1. Data flow — how information moves through systems (ingestion, transformation, serving)
2. Data modeling — organizing data into logical layers that serve multiple consumers
3. Platform integration — unifying analytics and data science workloads on shared infrastructure
4. Governance — managing data as discoverable, trusted, reusable assets
5. Pipeline design — intent-based (declarative) rather than implementation-based (imperative) definitions

These map directly to the kinds of questions [[01-projects/phdata/index]] clients ask: "How should we organize our warehouse layers?" (modeling), "How do we expose metrics consistently?" (governance + flow), "Should we consolidate on one platform?" (integration).

The Named DEDPs

The chapter previews several patterns explored in detail later:

• Dynamic Querying — ad-hoc querying without reprocessing; covered in [[06-reference/2026-04-04-dedp-dynamic-queries]]
• Stratified Data Flow Modeling — layered data organization (bronze/silver/gold, or raw/curated/aggregated)
• Open Data Platform / Lakehouse — unified analytics on open formats; connects to the Data Lake and DWH convergence in [[06-reference/2026-04-04-dedp-dwh-mdm-datalake-reverse-etl-cdp]]
• Asset-based Governance — managing data as products; relates to [[06-reference/2026-04-04-dedp-data-asset-reusability-pattern]]
• Declarative Pipelines — intent-driven architecture; evolution from imperative orchestration in [[06-reference/2026-04-04-dedp-etl-tool-comparisons]]

Why This Framing Matters

DEDPs transcend hype cycles. Tools come and go — Informatica to dbt to whatever is next — but the structural problems remain. The pattern vocabulary gives you a way to evaluate new tools against timeless requirements rather than getting swept into trend-based decisions.

This is the [[06-reference/concepts/systems-over-goals]] principle applied to data architecture: design for the recurring problem, not the tool of the moment.

Connections

• Sub-patterns (cache, reusability, workspace packaging) are DEPs covered in Chapter 5 — see [[06-reference/2026-04-04-dedp-cache-pattern]] and [[06-reference/2026-04-04-dedp-data-asset-reusability-pattern]]
• The materialized view / OBT / OLAP cube convergence in [[06-reference/2026-04-04-dedp-mv-obt-dbt-olap-dwa]] shows DEPs in action
• Semantic layer as logic encapsulation: [[06-reference/2026-04-04-dedp-semantic-layer-bi-olap-virtualization]]
• Data contracts as governance mechanism: [[06-reference/2026-04-04-dedp-data-contracts-schema-evolution]]