In short: an AI data quality checklist has six items — coverage, labels, duplicates, leakage, freshness and provenance. Most failed AI projects failed on one of them rather than on model choice. Data work is unglamorous, comes first, and sets a ceiling that no amount of model tuning can lift.
Teams spend weeks choosing between models and hours examining the data those models will consume. That allocation is backwards, because the model is a commodity decision and the data is the part nobody else has.
How we approached this
This describes established practice. RankBoast has not audited data programs and publishes no figures of its own — see our review methodology.
The six checks
| Check | The question | Failure looks like |
|---|---|---|
| 1. Coverage | Does the data include the cases you will meet in production? | Works in testing, fails on the edge cases that matter |
| 2. Labels | Are labels consistent between annotators? | A ceiling on accuracy nobody can explain |
| 3. Duplicates | Are near-identical records inflating apparent volume? | Overstated evaluation scores |
| 4. Leakage | Does any input reveal the answer? | Excellent test results, useless in production |
| 5. Freshness | Does the data still describe current reality? | Quiet, gradual degradation |
| 6. Provenance | Do you know where each record came from and may you use it? | A legal problem discovered late |
Leakage is the one that fools everyone
Leakage means information that will not exist at prediction time has crept into the training inputs. The classic form is a field populated only after the outcome is known — a resolution code, a closing date, a status flag updated on completion.
It is insidious because it produces excellent results. A model scoring far better than the problem should allow is the primary symptom. When results look too good, look for leakage before celebrating — and in any AI data quality checklist this is the check worth running twice.
Label consistency sets your ceiling
If two competent annotators disagree on a fifth of your examples, no model can exceed roughly that agreement level, because the target itself is inconsistent.
Measure it rather than assuming it. Have two people independently label the same two hundred examples and compute how often they agree. If agreement is poor, the fix is a clearer labeling guideline with worked examples for the ambiguous cases — not more data, and certainly not a bigger model.
AI data quality checklist: audit the tail
Datasets over-represent the common case, because the common case is what accumulates. The failures that damage trust happen on the unusual case: the unexpected language, the malformed input, the customer type that only appeared last quarter.
Practical method: list the categories you expect in production, count examples of each in your data, and look hardest at the categories with fewest. Then decide deliberately whether to gather more or to route those cases to a human.
Provenance is a legal control, not bookkeeping
For each source, record where it came from, when, under what terms, and whether it contains personal data. Do this while assembling the dataset, because reconstructing it later is close to impossible.
This matters more with generative systems than it did with traditional analytics, because a model may reproduce fragments of its inputs. Provenance also decides whether you can honor a deletion request, which is a legal obligation in many jurisdictions rather than a nice-to-have.
The same checks apply to retrieval systems
Retrieval-augmented systems have no training step, so teams assume data quality is irrelevant. The opposite is true: the corpus is the system. Superseded documents, three versions of one policy and drafts alongside approved copies all produce confidently wrong answers, and no prompt fixes them.
Audit the corpus before building — deduplicate, retire outdated documents, and decide which source wins when two disagree. See our RAG explainer for how retrieval failures present.
Common mistakes
Celebrating unusually good results. Look for leakage first.
Adding data to fix inconsistent labels. More noise does not average out to signal.
Evaluating on data containing duplicates of the training set. Flattering and meaningless.
Assuming retrieval systems need no data hygiene. The corpus is the product.
Recording provenance afterwards. It never gets done.
Auditing the average case. Failures live in the tail.
Where to start
- Before any project: provenance and coverage. Both are cheap and both stop expensive mistakes.
- Results look too good: leakage audit, immediately.
- Accuracy plateaued: measure inter-annotator agreement.
- Building retrieval: corpus deduplication and retirement first.
- Model degrading in production: freshness. Compare current inputs to your training distribution.
Verdict
Run the six checks before choosing a model, because they determine the ceiling the model will hit. Of the six, leakage and label consistency cause the most expensive surprises — one flatters you before production, the other caps you invisibly. An AI data quality checklist is dull work that decides whether the interesting work has any chance.
What we would need to test to say more
Quantifying how much each data problem costs in accuracy would require controlled experiments introducing each fault into a clean dataset and measuring the effect. We have not done that and publish no figures.
Sources and methodology
This article describes documented practice rather than reporting tests. RankBoast has not audited any organization’s data or AI programs, accepts no vendor sponsorship, and publishes no benchmark or savings figures of its own. Research and drafting were AI-assisted. Errors are handled under our corrections policy.
Source links
Join the discussion
Add useful context, ask a focused question or share relevant experience. Comments are moderated to protect readers from spam and promotional links.
Leave a thoughtful comment