DEDP 5.1 — Data Engineering Patterns (Intro)
The framing chapter for DEPs — the middle layer of the CE → Pattern → Design Pattern hierarchy. DEPs are "generally repeated, identifiable designs or practices" in data engineering. They sit between raw convergent evolutions (specific tools) and prescriptive design patterns (best-practice implementations).
The Questions DEPs Answer
The chapter frames DEPs through practical questions practitioners face daily:
| Question | Pattern Area | Related Technologies |
|---|---|---|
| How do I cache data for BI? | Caching | dbt, materialized views, OLAP, data virtualization |
| How do I orchestrate pipelines? | Orchestration | Cron, Python frameworks, Airflow, custom solutions |
| How do I manage master data without duplication? | MDM / Data Integration | Fuzzy joins, entity resolution, golden records |
| How do I package my data stack? | Workspace Packaging | Docker, Kubernetes, cloud platforms |
| How do I share data across teams? | Data Sharing | File exports, delta live tables, open standards |
| How do I combine stateful and stateless assets? | Asset Integration | Infrastructure as Code, declarative data stacks |
Each question points to a pattern; each pattern has multiple convergent evolution implementations. This is the practical value of the framework — it gives you a name for what you are doing, which means you can find prior art, compare approaches, and avoid reinventing solutions.
Position in the Book
This chapter is a gateway. The specific DEPs covered in subsequent chapters:
- [[06-reference/2026-04-04-dedp-cache-pattern]] (5.2) — the canonical pattern example
- [[06-reference/2026-04-04-dedp-data-asset-reusability-pattern]] (5.3) — composing and reusing data assets
- [[06-reference/2026-04-04-dedp-de-workspace-packaging]] (5.4) — packaging and deploying data workspaces
Why DEPs Matter for Consulting
For [[01-projects/phdata/index]] engagements, DEPs are the conversation layer. Clients do not think in design patterns — they think in problems. "How do I make my dashboards faster?" is a caching question. "How do I let teams work independently?" is a workspace packaging question. Naming the pattern unlocks the design pattern, which unlocks the implementation.
This is [[06-reference/concepts/analytics-as-craft]] at the architectural level: craft is knowing what pattern you are implementing and choosing the right instantiation, rather than reaching for the tool you happen to know.
Connections
- The hierarchy that DEPs belong to: [[06-reference/2026-04-04-dedp-intro-dedp]]
- Convergent evolution theory behind patterns: [[06-reference/2026-04-04-dedp-convergent-evolution]]
- The flowchart mapping CEs to DEPs to DEDPs: [[06-reference/2026-04-04-dedp-ce-intro]]
- Design patterns (the layer above DEPs): [[06-reference/2026-04-04-dedp-design-patterns-intro]]