06-reference

dedp intro dedp

Fri Apr 03 2026 20:00:00 GMT-0400 (Eastern Daylight Time) ·book-chapter ·source: https://www.dedp.online/part-1/2-overview-dedp/_intro-dedp.html ·by DEDP / Simon Späti

DEDP 2.1 — Introduction to Data Engineering Design Patterns

The conceptual anchor for the entire book. This chapter establishes the vocabulary and hierarchy that everything else builds on: convergent evolution produces patterns, patterns inform design patterns. If you only read one framing chapter, this is it.

The Hierarchy

The book’s core intellectual move is borrowing convergent evolution from biology — when distinct evolutionary paths arrive at similar outcomes independently. Birds evolved feathers, bees evolved exoskeletons; both achieved flight. Data engineering does the same thing: different eras, different tools, same underlying problem solved the same way.

This produces a three-level hierarchy:

  1. Convergent Evolutions (CEs) — independent technologies that solve the same problem (e.g., materialized views and OBTs both cache query results)
  2. Data Engineering Patterns (DEPs) — the high-level repeated practice those CEs reveal (e.g., caching). See 06-reference/2026-04-04-dedp-dep-intro
  3. Data Engineering Design Patterns (DEDPs) — specific, implementable best-practice solutions for recurring problems (e.g., the 06-reference/2026-04-04-dedp-dynamic-queries pattern)

Key Definitions

Pattern: “A repeated, identifiable design, procedure, or practice” appearing across different contexts. Patterns have observable predictability and repeatability — they are discovered, not invented.

Design Pattern: Reusable, flexible approaches to recurring design challenges. The concept traces directly to the Gang of Four’s Design Patterns: Elements of Reusable Object-Oriented Software (1994), which categorized solutions into creational, structural, and behavioral types.

DEDP: Domain-specific design patterns applied to data engineering challenges within the Data Engineering Lifecycle.

Why This Matters

The distinction between DEPs and DEDPs is the book’s most important conceptual contribution. DEPs are descriptive — they name what practitioners already do (ETL, pipelining, caching, streaming). DEDPs are prescriptive — they tell you how to implement those patterns well.

This maps directly to 06-reference/concepts/systems-over-goals: patterns are the system, individual tool choices are the goals. When you recognize that materialized views, OBTs, and OLAP cubes are all instances of the cache pattern, you stop debating tools and start designing systems. That is the shift this book enables for 01-projects/phdata/index consulting engagements.

Connections