The Downfall of the Data Engineer — Conformed Dimensions and Pipeline Constipation

Maxime Beauchemin (creator of Airflow and Superset) on the systemic challenges that erode data engineering effectiveness.

Key concepts

Conformed dimensions and metrics

The quest to achieve conformed dimensions and conformed metrics — shared, agreed-upon definitions across the organization — is as relevant as ever. Trust in data requires consistency and alignment, but in large organizations where hundreds of people generate data, consensus-seeking is challenging or impossible in a timely fashion.

Pipeline constipation

When an organization values stability over accuracy, change becomes scary. If data engineers’ incentives are geared toward stability, they learn quickly: the best way to not break anything is to not change anything. This leads to pipeline constipation — systems that calcify and resist necessary evolution.

Data silos

Silos naturally spawn as projects start, teams drift, and acquisitions occur. The pejorative term “data silo” describes heterogeneous analytics scattered across platforms using incompatible references.

Iteration cycle pressure

When idle time between iteration cycles is counted in hours, it becomes tempting to work around the clock to keep “plates spinning.” When 5-10 minutes at 11:30pm saves 2-4 hours the next day, unhealthy work-life balance follows. The fix: tighten iteration cycles where possible.

Connects to data challenge is organizational, Uber data culture, DevOps and modern data experience, systems over goals.

Open questions

• How do you incentivize data engineers toward accuracy without creating pipeline constipation?
• Is data mesh a viable solution to the conformed dimensions problem, or does it just distribute the silos?