ayushdoesdev
← All work

Independent lab · not an employer deliverable

DataForge

Data Pipeline and Quality Control

Inspect a sample ingestion DAG, quality gates, schema drift, and controlled reruns without a live warehouse.

Jump to interactive demo

Problem

Pipeline UIs often show green boxes while quality issues hide in rejected rows. DataForge treats contracts, checks, and reruns as the product—not just task status.

Interactive demo

Local pipeline and sample records. Not connected to a warehouse or scheduler.

DataForge · pipeline workbenchSample batch · local DAG

Pipeline graph

Selected node

Apply enabled quality checks before load.

Inputs: previous stage rows · Outputs: 2 accepted

Run history

2026-10-28T06:08Z · 9 min · processed 0 · rejected 4. Validation failed on duplicate keys and an invalid timestamp.

Quality checks

Sample evaluation: 2 accepted / 5 rejected with the enabled gates.

  • evt-02 · c-104 · duplicates
  • evt-03 · (null) · nulls
  • evt-04 · c-221 · range
  • evt-05 · c-308 · timestamps
  • evt-06 · c-441 · referential

Schema drift

Analytics extract expects `customer_id`. Source now emits `acct_id` on the sample contract.

Analytics node is skipped until the contract is resolved.

Architecture

A five-stage DAG (ingest → normalize → validate → warehouse → analytics) executes over bundled sample records. Quality checks score the same rows the inspector shows. Schema drift is a named sample change with an explicit resolution. This is a client-side orchestration sketch, not Airflow.

Source batch → normalize types → run enabled quality checks → load accepted rows → analytics extract. Failures isolate at the validating stage unless retried.

Decisions

  • — Keep quality checks configurable so disabling a gate changes the accepted-row count.
  • — Model schema drift as a reviewable event rather than a silent column rename.
  • — Make retry, rerun, and backfill update the same run history the graph reads.

Trade-offs

  • — Failing a pipeline on quality protects downstream models and it delays availability.
  • — Skipping a check restores throughput and it can load duplicates or null keys.
  • — Backfills repair history and they can double-count if loads are not idempotent.

Failure modes

  • — A null required column trips the nulls check and rejects the row.
  • — A renamed column blocks analytics until the contract is accepted or pinned.
  • — Retrying a failed validate stage does not rewrite already-loaded warehouse rows.

Evaluation

Counts of processed and rejected records are computed from the sample batch and the enabled checks. They are demo calculations.

Production extensions

Proposed — not implemented in this portfolio.

  • — Compile the DAG to Airflow or Dagster with real sensors.
  • — Persist data contracts in a schema registry.
  • — Add column-level lineage across warehouses.

Python · Apache Airflow · SQL · PostgreSQL · data quality · lineage