Self-healing ETL engine
When a data source renames or retypes a column, this pipeline maps the column back to the expected one if it is confident, holds the batch for review if it is not, and logs every decision.
- Python
- Polars
- PostgreSQL
- dbt
How it works
- Incoming filerenamed or retyped columns
- Validateagainst the master schema
- Match columnsname, history, meaning
- Confidence floorsure enough to remap?
- Standardizefixed rules
- SinkPostgres, Parquet, CSV
- Review queueheld for a person
Decisions that matter
Matching in layers
A column is matched by exact name first, then by what it was mapped to before, then by spelling, then by meaning. The layers feed one confidence score, and a check on the data itself can veto a match whose values do not look right.
Stop rather than guess
A mapping is accepted only at a confidence of 0.92 or higher. If a required column is left uncovered, or two source columns land on one target, the whole batch is quarantined for review instead of loaded.
Every decision can be replayed
Each mapping decision is stored with its confidence in an audit trail. A quarantined batch can be reviewed and replayed once a person fixes the mapping, and accepted mappings are remembered, so the same drift is not worked out twice.
Tested by breaking it
A chaos harness renames columns, changes types and injects nulls into clean files. It then checks that the output still matches the master schema and keeps every row.
Limits
The core path of checking, remapping, holding back and auditing is the part marked implemented. The connectors and document features around it are still experimental.