Data engineering · Architecture

Follow the data lifecycle

A data problem is easier to diagnose when you trace the whole journey instead of staring at the failing destination.

Inspired by Fundamentals of Data Engineering by Joe Reis and Matt Housley. This is an original explanation, not a reproduction of the text.

The idea

A dashboard is only the final stop in a longer journey. Data is created in a source, moved, stored, changed, and eventually delivered for use. A defect visible at the end may have entered at any earlier stage.

Think in lifecycles before thinking in products. For each stage, ask what the data means, who owns it, how quickly it must move, what can fail, and what evidence would reveal failure. Also look across the journey: security, governance, orchestration, architecture, and software quality are not isolated steps. They support every step.

Make it concrete

A revenue dashboard suddenly drops by 12%. Rebuilding the chart will not help if a checkout service stopped emitting one event type. The team traces one order from its source through ingestion, storage, transformation, and serving. Row counts are healthy until ingestion, where a schema change caused records to be rejected. The lifecycle points the investigation toward the boundary where reality first diverged.

Keep this: When data looks wrong, walk upstream stage by stage and find the earliest point where it became wrong.

Try it

Choose one report you depend on. Sketch its path from the system that creates the data to the person who uses it, then mark one failure signal at each stage.