The pattern is consistent enough to be predictable. A proof of concept works impressively on an extract somebody prepared by hand. The same system meets production data and the answers degrade, the exceptions multiply, and the project quietly loses its sponsor. The model was never the problem.
"AI-ready data" gets used as a slogan. This article makes it concrete: what the phrase actually requires, how to assess where you stand, the signs that you are not there yet, and a phased way to close the gap without launching a two-year data programme first.
Quick answer: what does AI-ready data mean?
AI-ready data is data that is discoverable, accurate, well-described, governed and accessible under the requesting user's permissions — reliably enough that an AI system can use it without a person curating it first. In practice that means four properties hold: someone owns the joined view across systems, quality is measured at source rather than cleaned by hand, lineage can be reconstructed, and entitlements travel with the data so retrieval can filter on them.
Key takeaways
- A pilot has these properties by accident. Production requires each of them to be engineered.
- Ownership of the cross-system view is the most common blocker, and it is organisational rather than technical.
- Entitlements must be enforced at retrieval, not applied to results afterwards — a model that has read something can reproduce it even when the citation is stripped.
- Scope readiness to one use case rather than the whole estate. Estate-wide data programmes stall; use-case-scoped ones finish.
- Unstructured content is where most enterprise knowledge lives and where most readiness work is skipped.
- The prompt and retrieval log is a new store of regulated data that the project itself creates.
Why data quality decides AI outcomes
Traditional reporting is forgiving of imperfect data in a way AI is not. A dashboard with a few wrong records still shows a usable trend, and an analyst who knows the data applies mental corrections without noticing. An AI system does neither. It treats what it retrieves as true, states it confidently, and offers no signal that the underlying record was stale.
Four dimensions matter, and each should be measured against the specific use case rather than in the abstract:
- Completeness. What proportion of records carry the fields this use case depends on? Not all fields — the ones the system will actually read.
- Validity. Do values conform to the rules that are meant to hold? Dates in range, codes in the reference set, amounts with a plausible sign.
- Consistency. Does the same entity carry the same identifier across systems? This is usually where the real cost sits.
- Timeliness. How stale is the data when the system reads it, and does the use case tolerate that?
A customer address that is sixty per cent complete is fatal for a logistics assistant and irrelevant for a churn model. That is why generic quality programmes stall and use-case-scoped ones finish: the first tries to fix everything, the second fixes what one deployment needs and then reuses the plumbing.
Data silos and the ownership problem
Most enterprises have clear ownership of individual systems and none at all of the view that spans them. The CRM has an owner. Billing has an owner. The question an AI assistant is asked — "which customers are at risk and why" — needs both, plus support history, and the joined answer belongs to nobody.
When that joined view is wrong, three teams each correctly report that their own system is fine. Nobody is accountable for the discrepancy, so it is never resolved, and the AI system inherits it.
The fix is organisational. Name an owner for each data product that spans systems — a person accountable for its definition, its quality and its access policy. This is the core of data product thinking and it is worth adopting even if you adopt nothing else from that vocabulary. It costs nothing but a difficult conversation, and almost every subsequent technical decision depends on it.
Data governance, security and access
In a pilot, everyone on the project sees everything. In production, different users are entitled to different subsets — and if the AI system does not honour that, it becomes a permission bypass with a friendly interface.
This has a specific architectural consequence: entitlements have to be expressible as data, attached to rows or documents, so retrieval can filter on them before the model sees anything. Applying permissions to the output afterwards does not work. A model that has read a document it should not have can restate its content in a summary even when the citation is removed.
Two further points that are routinely missed:
- An embedding index inherits every permission mistake in its source. Content shared too broadly years ago was theoretically reachable and practically invisible; indexed, it is one question away. Audit source permissions before building the index, not after the first surprise.
- The prompt and retrieval log is regulated data. It contains whatever was retrieved to answer each question, so it inherits the same classification, retention and residency obligations as the source systems. It is created by the project and frequently assigned to nobody.
We go deeper on the control set in securing the enterprise data your AI models touch, and on how governance itself has to change in data governance in the age of AI agents.
Metadata, lineage and master data
Metadata
An AI system choosing between two similarly named tables has no way to know which one the business actually uses unless something tells it. Descriptions, ownership, freshness and classification recorded in a catalogue are what turn a data estate into something a machine can navigate. This is also the least glamorous work in any readiness programme, and the most reliably deferred.
Lineage
When a system produces a number that surprises someone senior, the first question is where it came from. If answering requires archaeology across notebooks and scheduled jobs, confidence collapses faster than any accuracy metric can restore it. The minimum viable version is a documented path from source system to consumed dataset, with transformations expressed as code in version control rather than as manual steps. Automated column-level lineage is valuable at scale but it is an optimisation of something that must first exist.
Master data
If "customer" means something different in three systems, an AI system joining across them produces confidently wrong answers. Master data management is unfashionable and it is frequently the actual blocker. It does not need to be solved estate-wide — it needs to be solved for the entities the use case touches.
Structured, unstructured and real-time data
Most enterprise knowledge does not live in a database. It lives in documents, tickets, email threads, meeting notes and slide decks. Readiness work concentrates on structured data because that is where the tooling is comfortable, and then the assistant is asked a question whose answer only exists in a policy document nobody has classified.
For unstructured content, readiness means:
- Findable. Stored somewhere indexable, not on a personal drive.
- Current. Superseded versions removed or marked, because a retriever cannot infer which of three contradictory documents is in force.
- Classified. Sensitivity labelled, so retrieval can filter.
- Structured enough to chunk. Documents with real headings retrieve far better than a wall of text, which is a content standards question as much as a technical one.
Real-time data raises a different question: does the use case genuinely need it? Real-time pipelines cost significantly more to build and operate than scheduled ones. Many assistants described as needing live data are perfectly served by hourly refresh. Establish the actual tolerance before designing for the hardest case.
Data architecture and platform choices
Platform choice matters less than the properties above, but it is not neutral. The practical question is which shape supports both analytical reporting and AI workloads without maintaining two diverging copies of the truth — the difference between a lake, a warehouse and a lakehouse, which we work through in data lake, warehouse or lakehouse.
Two considerations specific to AI:
- Retrieval needs a permission dimension. Whatever the platform, the index must be able to filter on the requesting user's entitlements. Retrofitting this means rebuilding the index and revalidating every answer.
- Retrieval-augmented generation is a data problem, not a model problem. Chunking that respects document structure, hybrid keyword-and-vector search, reranking and metadata filters are what determine answer quality. Teams that skip reranking usually conclude the approach does not work for them, when what did not work was naive similarity search. Our comparison of RAG and fine-tuning covers when each is appropriate.
Is your data ready for AI? A readiness checklist
Answer these for one specific intended use case, not for the estate:
- ☐ A named person owns the joined view this use case needs
- ☐ Completeness of the required fields is measured, not assumed
- ☐ Validity rules exist and are checked at source
- ☐ The same entity carries a consistent identifier across the systems involved
- ☐ Data freshness meets what the use case actually tolerates
- ☐ Lineage from source to consumed dataset is expressed in version-controlled code
- ☐ Sensitivity classification is applied to every source in scope
- ☐ Entitlements exist as attributes that retrieval can filter on
- ☐ Source permissions have been audited before any index is built
- ☐ Unstructured content in scope is findable, current and structured enough to chunk
- ☐ A catalogue records what each dataset is, who owns it and how fresh it is
- ☐ Retention and residency obligations are identified for the prompt and retrieval logs
10 signs your organisation is not AI-ready
- The proof of concept ran on a spreadsheet somebody prepared by hand.
- Nobody can say who owns the joined view across the systems involved.
- Two teams produce different numbers for the same measure and both are defensible.
- Data quality is described as "generally good" rather than as a measured figure.
- Access is granted at platform level, so anyone with the tool sees everything in it.
- The answer to "where did this dataset come from" is a person who remembers.
- Key documents live on personal drives or in email threads.
- Nobody can say which of three similar policy documents is currently in force.
- Sensitivity classification exists as a policy but not as labels on actual content.
- The AI pilot's prompt logs have no owner, no classification and no retention period.
Three or four of these is common and not fatal. It does mean the next investment should be the foundation rather than another model.
How to build an AI-ready data foundation
A phased approach, scoped to a use case rather than the estate.
| Phase | What you do | Exit criteria |
|---|---|---|
| 1. Scope | Pick one use case and list the systems and content it touches | A defined boundary, not the whole estate |
| 2. Own | Name an accountable owner for each data product in scope | A person named against each, not a team |
| 3. Classify | Label sensitivity and record who may see each source | Every in-scope source has a decision recorded |
| 4. Access | Make entitlements filterable attributes; audit source permissions | Retrieval can enforce permissions before generation |
| 5. Measure | Instrument quality on the fields the use case needs | A trended number, not an opinion |
| 6. Trace | Move transformations into version-controlled code | Any output can be traced to its source |
| 7. Extend | Reuse the plumbing for the next use case | Second use case costs materially less than the first |
The order is deliberate. Ownership first, because everything after it needs somebody to decide. Access model second, because retrofitting entitlements means rebuilding indexes and revalidating answers. Quality and lineage follow, and they are cheaper once ownership exists.
Common mistakes
- Launching an estate-wide data programme before any AI use case exists. It will run for two years and be overtaken.
- Hand-cleaning data for the pilot. It hides both the true state of the data and which problems actually matter.
- Applying permissions to output instead of to retrieval. A leak that is invisible in testing, because testers usually have broad access.
- Indexing first and auditing permissions later. The index reproduces every sharing mistake at machine speed.
- Ignoring unstructured content because the tooling is less comfortable, when that is where the answers live.
- Designing for real-time when the use case tolerates an hourly refresh.
- Leaving prompt logs unclassified. They quietly become the most concentrated store of sensitive data in the estate.
Conclusion
When an AI pilot works and production does not, the model is almost never the reason. The pilot ran on data with four properties that were true by accident — a single owner, hand-checked quality, remembered lineage and uniform access. Production requires each to be true by design.
You do not need a finished data platform to start. You need those four properties to hold for the one use case you intend to deploy, which is a far smaller scope than an estate-wide programme and is achievable in weeks rather than years. Establish ownership and the access model first, because they are the cheapest to put in place and the most expensive to retrofit.
Frequently asked questions
What does AI-ready data actually mean?
Data that is discoverable, accurate, well-described, governed and accessible under the requesting user's permissions, reliably enough that an AI system can use it without human curation. Concretely, four properties hold: a named owner for the view that spans systems, measured quality on the fields the use case needs, lineage reconstructible from version-controlled code, and entitlements expressed as attributes retrieval can filter on.
Do we need a complete data platform before starting with AI?
No, and attempting one is how these programmes stall. You need the four readiness properties to hold for the specific use case you intend to deploy — a far smaller scope than an estate-wide platform. Start with ownership and the access model, because they cost least to establish and most to retrofit, then reuse the plumbing for the next use case.
Why do AI projects fail because of data rather than models?
Because the pilot ran on a hand-cleaned extract from one system that everyone on the project could see in full, and production data has none of those properties. It has multiple owners, uncontrolled quality, lineage nobody can reconstruct and access that varies by user. Swapping the model does not address any of those, which is why teams that try it get the same result twice.
How do we stop an AI assistant surfacing data a user should not see?
Apply the requesting user's entitlements inside the retrieval query rather than to the results afterwards. Carry group membership or an equivalent access key as filterable metadata on every indexed item. Filtering after generation does not work: a model that has read a restricted document can restate its content in a summary even once the citation is stripped.
Does unstructured data need to be AI-ready too?
Yes, and it is where most enterprise knowledge lives. Readiness for documents means findable rather than on a personal drive, current with superseded versions removed or marked, classified for sensitivity so retrieval can filter, and structured with real headings so it chunks well. Readiness programmes concentrate on structured data because the tooling is comfortable there, then find the answer only existed in an unclassified policy document.
What is the first thing to fix if our data is not ready?
Ownership of the view that spans systems. Individual systems have owners; the joined answer an AI use case needs belongs to nobody, so when it is wrong each team correctly reports their own system is fine and the discrepancy is never resolved. It costs nothing but a difficult conversation, and nearly every subsequent technical decision depends on it.
Are AI prompt and retrieval logs sensitive data?
Yes, and often the most concentrated store in the estate, because each entry contains whatever was retrieved to answer the question. They inherit the same classification, retention and residency obligations as the source systems. Assign an owner and a retention period at design time — created without either, they accumulate quickly and are discovered during an assessment rather than before one.
