Data platform

Target state

In short: Agents answer from governed company data that respects permissions, carries its source, and is only as fresh as decisions need.

GroundingGrounding. Making an agent answer from trusted company sources it can point to, and refusing when the evidence is weak. (answering from trusted company sources) is a governed product, not a side effect of a data lake. The data agents ground on is well modelled. Wherever the stakes demand a deterministicDeterministic. Always gives the same answer for the same input by following fixed rules. The opposite of probabilistic, where the answer can vary. answer (the same answer every time), the business definitions behind it are governed. Indexes are permission-aware, and retrievalRetrieval. Finding the few relevant pieces of company information for a task and placing them in front of the model. is filtered on the caller's live identity before results are returned. Freshness is fed by change data capture (CDC)Change data capture (CDC). Streaming each change in a database as it happens, so copies, including agent indexes, stay current., which streams each database change as it happens. It is sized to how often decisions are made, not to fashion. ProvenanceProvenance. The record of where a piece of information came from, carried with it so every answer can be traced back to its source. runs from every answer back to the exact passage it came from. Catalogues supply ownership, lineageLineage. The chain of steps data went through to reach its current form., and definitions as contextContext window. The model's working memory for one task. It is limited in size, so what goes in and what gets left out is a design decision. for agents, not as a portal humans visit. EmbeddingsEmbedding. A numeric fingerprint of a piece of text that lets a computer find similar meaning quickly. Embeddings are derived data and inherit the protections of what they came from. (numeric fingerprints of text) are treated as recoverable data, and they carry their sources' classificationClassification (data). A label for how sensitive data is. Anything derived from data inherits the strictest label of its sources.. Anything derived from several sources inherits the strictest classification among them.

layer · architecture

Grounding as a governed product

Permissions crawl into index metadata and retrieval is pre-filtered on the caller's live identity; post-filtering is the anti-pattern.

1. 01System

   System

   Source systems

   Documents, records, events with native ACLs

2. 02Control

   Control

   Permission crawl

   ACLs into index metadata; ReBAC pre-computation

3. 03Evidence

   Evidence

   Governed index

   Version-stamped vectors; purpose-scoped where stakes demand

4. 04Control

   Control

   Pre-filtered retrieval

   Caller's live identity inside the query

5. 05Control

   Control

   Semantic contract

   Deterministic answers or refusal for high-stakes numerics

6. 06Agent

   Agent

   Agent context

   Answer with span-level source provenance

Embeddings are recoverable text; vectors inherit source classification and erasure obligations.

Diagram description: ACL-aware grounding pipeline from source systems through permission crawl into index metadata, pre-filtered retrieval on live identity, and span-level provenance, with fail-close branch. The map contains Source systems: Documents, records, events with native ACLs; Permission crawl: ACLs into index metadata; ReBAC pre-computation; Governed index: Version-stamped vectors; purpose-scoped where stakes demand; Pre-filtered retrieval: Caller's live identity inside the query; Semantic contract: Deterministic answers or refusal for high-stakes numerics; Agent context: Answer with span-level source provenance. Its connections are sources to crawl; crawl to index for fail-close on sync error; index to retrieval; retrieval to agent; semantics to agent for numeric questions. Important boundary: Embeddings are recoverable text; vectors inherit source classification and erasure obligations.

Reviewed 2026-08-20·Sources:r02-data-platform findings

Mechanisms

ACL-aware retrieval is deterministic security applied to grounding

In short: Check who is asking before searching, so an agent can never surface a document its user could not open.

ACL-aware retrievalACL-aware retrieval. Retrieval that checks the requester's permissions before returning information, so an agent cannot surface documents its user could not open. checks the access control list (ACL), the record of who may open each document, before returning results. The shipped pattern: crawl source permissions into index metadata, then filter inside the vector query on the caller's live identity. Document-level directory ACLs are native on the major search platforms. Connector-crawled permission maps ship with fail-closeDegraded mode. What a system does when a part fails. For agents, that means falling back to a human queue rather than guessing. behaviour: on an error, they deny rather than allow. Relationship-based authorisation engines of the Zanzibar class pre-compute the authorised objects for the hard cases. Filtering after the search is the anti-pattern. It leaks result counts and starves the result set. Four named failure modes survive a correct implementation and must be designed for. First, permission-sync lag: a revoked permission takes effect at the next crawl, not at the click. Second, group-membership drift. Third, documents with no ACL defaulting to public. Fourth, inherited oversharing, where agents amplify a decade of accumulated permission sprawl. Vendor-published estimates put roughly 16 percent of business-critical data in an overshared state. The mitigation shipped by one major vendor is a scoped-search allowlist that its own documentation calls a stopgap. Hygiene comes before retrieval. No retrieval layer fixes wrong ACLs.

Embeddings are recoverable data

In short: The numeric form of a document can often be turned back into its text, so it needs the same protection as the document.

Exact reconstruction of short inputs from their embeddings has been demonstrated at 92 percent. The result is vec2text, presented at the Empirical Methods in Natural Language Processing conference (EMNLP) in 2023 and reproduced in 2025. The architectural consequence is not optional. Vectors inherit the source's classification, ACLs, residencyData residency and sovereignty. Rules about which country or region data may be stored and processed in, including the copies created for agents. rules, and privacy obligations. ErasureErasure cascade. When a record is deleted, every copy derived from it (embeddings, memories, traces, test data) is deleted too. cascades into vectors and every derived artifact: delete the record, and every copy made from it goes too. This is the reasoning behind the named risk in the OWASP (Open Worldwide Application Security Project) application list. Treat an index as a copy of the data, because that is what it is.

Semantic contracts convert silent errors into refusals

In short: Fixed business definitions turn a confidently wrong number into either the right number or an honest refusal.

For high-stakes numeric questions, the value of a governed semantic layerSemantic layer. A shared set of business definitions (what counts as revenue, or an active customer) that keeps agents and reports using the same meaning. (a shared set of business definitions) is not a few points of accuracy. It is failure-mode conversion: a wrong number becomes either a refusal or a deterministic answer. Ontology-checked querying moved accuracy from 54 percent to 72 percent. It also produced explicit "I do not know" responses on about 8 percent of questions, which is the property that matters. A vendor's paired benchmarkBenchmark. A public test used to compare models or agents. Useful for orientation; scores can be gamed, so they are claims rather than measurements. compares a semantic layer against text-to-SQL, where the model writes Structured Query Language (SQL) database queries directly. It reports 98 to 100 percent accuracy through the semantic layer on the modelled scope, against 84 to 90 percent for text-to-SQL. The semantic layer covered 72.7 percent of the full question set [vendor]. A semantic-document approach added 17 to 23 points across three frontier models [vendor]. The honest counter-camp reaches comparable accuracy through disciplined data modelling alone. It argues that good modelling is the semantic layer. The floor everyone should know: on the hardest public text-to-SQL benchmark, frontier models score between single digits and the low twenties, in percent. On its predecessor they score above 90 percent.

The retrieval substrate ladder

In short: Start with the simplest storage that fits the number of documents, and move up only when measurements say so.

The ladder has rungs with thresholds, because "which vector database" is the wrong first question. Below roughly 100,000 vectors, brute-force search is enough. Postgres with the pgvector extension serves roughly 1 to 10 million vectors. At 1 million it answers in under 20 milliseconds with above 95 percent recall (the share of relevant items it finds). Index-build pain appears around 2 million, and partitioning becomes necessary past 5 million. Scale-out Postgres extensions reach roughly 50 million. Published comparisons show large advantages at 99 percent recall against a managed peer, at lower cost [vendor]. The advantages are in p95 latencyLatency. How long a request takes to come back. (the time within which 95 percent of queries return) and in queries per second (QPS). Dedicated or managed engines earn the top rung for extreme scale, heavy write rates, or filter-heavy multi-tenancy (many tenants on one index). Take 100 million vectors with filters that select 20 percent of the data. There, Postgres p99 latency (the time within which 99 percent of queries return) runs several times that of a purpose-built engine. Two operational cautions decide real migrations. Index builds consume tens of gigabytes of memory (RAM) for hours. Filtered search has recall cliffs: past certain filters, it suddenly misses relevant items. Vector storage on object stores has reached general availability at up to 90 percent lower cost [vendor]. That tilts the economics toward more, smaller, purpose-scoped indexes rather than one large one.

Freshness is a cost decision, not a virtue

In short: Keep data as fresh as the decisions need, because instant updates cost far more than near-instant ones.

Change data capture feeding incremental indexing delivers freshness measured in minutes at close to batch cost. One documented like-for-like requirement priced at about $400 per month as a five-minute batch, against about $7,600 per month as streaming. Streaming commonly runs an order of magnitude (roughly ten times) above batch. Streaming is earned by decision cadence: fraud, operational triage, inventory. The failure mechanism to design against is specific. Similarity ranking ignores time, so a superseded document outranks its current replacement unless freshness or validity is part of the ranking.

Re-embedding is the hidden line item

In short: Changing the model behind the fingerprints means redoing every document, which costs real money and needs a plan.

Model upgrades force a re-embed of the full corpus. Reference costs run from hours on multi-GPU (graphics processing unit) hardware to four figures through an APIAPI. A defined way for software to talk to other software. Agents reach systems through governed APIs rather than through screens. (application programming interface). The operational discipline has four parts. Version-stamp every vector. Run blue/green index migrations: build the new index beside the old one and switch over only when it passes. Evaluate embedding models quarterly. Migrate only for evalEval (evaluation suite). A set of test tasks with known right answers, written by people who know the work, used to check an agent before and after any change. gains (gains on test tasks with known right answers) above a stated threshold, and 5 percent is the working figure. Cross-model conversion, published at roughly 100 times cheaper than re-embedding, is worth testing before any large migration. Re-embed changed chunksChunking. Splitting documents into pieces small enough for retrieval. How you split changes what the agent finds. (the pieces documents are split into) continuously, and keep a periodic full re-index as the recovery baseline.

Design decisions

• Semantic-layer product vs governed views (CD-8): decided per use case, by failure mode rather than by accuracy points. The binding variable is curatedCuration. Preparing data for a specific purpose before agents use it, with a named owner responsible for its quality. business context.
• Dedicated vector database vs the simplest substrate (CD-9): climb the ladder above one rung at a time. Prefer the vector capability bundled with the platform where the corpus already lives. Governance and ACL inheritance then come free.
• Data mesh vs governed central substrate (CD-10): mesh ideas on a governed substrate. Every shipped ACL-sync architecture assumes a governed central index.
• Real-time vs batch grounding (CD-12): CDC is the default. Streaming is earned by decision cadence.
• One universal index vs use-case-scoped (CD-11): two layers. The detail is resolved on R14, the agent data engineering page.

Cross-cutting concerns

Evidence and limits

The vec2text reconstruction result, the ontology-checked accuracy figures, and the substrate benchmarks are peer-reviewed or reproducible. The semantic-layer coverage comparisons and the substrate performance claims are vendor-published and flagged as such. The oversharing percentage is secondary-sourced and carried with that status. Two refusals. First, a widely circulated projection says that a majority of agentic analytics projects relying solely on tool protocols will fail by 2028. It is secondhand, with no located primary source, and this guide excludes it. Second, the substrate ladder's rung thresholds are working figures from published benchmarks, not guarantees for your corpus, so measure before you migrate. Re-verify quarterly: platform vector general-availability states, semantic interchange specification adoption, and embedding model releases that would trigger a re-embed decision.

The research behind this page

• Data platform findings
• Research brief
• Vendor landscape
• Sources