A green pipeline run does not prove the pipeline produced the right records.
Data systems can fail quietly. A job finishes, output tables still contain rows, reports still render, and no alert fires. From the outside, the release candidate looks healthy.
In one release candidate, a pipeline update touched several levels of the workflow and should have increased the result count. Standard validation did not show a failure. Before release, I added a check that asked a narrower question: did the updated tool write fresh results during this run?
It had not. Existing records masked the missing output, and the team did not yet know the full expected result set for the new tool.
Project note
Problem: A pipeline change should have increased the result count. Standard validation saw existing outputs and no failed job, so it could not prove the updated tool had contributed new data.
Action: I added a freshness check before release and compared the latest analysis timestamp against the current run.
Result: The check showed that one updated tool had stopped writing new records. The job did not crash, older database rows made the output look complete, and the release stopped before stale behavior reached production.
Lesson: A passing pipeline can still carry old data forward. Data QA needs freshness, provenance, and count checks, not only job status and output presence.
Why it matters
Stale data defects do not look like normal failures. You may see no crash, no empty output, and no obvious red flag.
If older records remain in the database, a broken step can disappear inside a mostly complete result set. Teams working on pipelines, analytics, reports, or scientific workflows need to prove which component produced the output and when it produced it.
What teams should check
Use these checks when a release depends on similar behavior.
- Did the updated tool write records during this run?
- Did timestamps change where the change should have produced new output?
- Can each result identify the producing step or tool version?
- Did the result count move in the expected direction?
- Could existing database rows hide missing new records?
- Can the tool finish without producing output?
- Do logs distinguish ran from produced expected records?