When AI Inherits Everyone's Permissions

The model is almost never the thing that leaks. The leak was configured months earlier, on a screen where somebody had to choose which identity the AI system would use to read the company's files — and chose the one that could read all of them.

A regional account manager opens the company's internal assistant before a renewal call and asks it something entirely ordinary: what should she know about this customer before she dials. The answer comes back in seconds, well organized and genuinely useful, and buried in the third paragraph is the pricing floor the deal desk set for that account, followed by a clause-level summary of a contract dispute that legal has been handling quietly for a month. Both facts are true. Both live in systems the company owns. Neither is something her role has ever granted her the right to see, and she did nothing unusual to obtain them — she asked a normal question in plain language and the system answered it well. If anyone eventually notices, the incident report will be written up as an AI problem, and that framing will be wrong in a way that matters, because it will send the organization looking for a fix in the place where the failure did not occur.

Nothing in that exchange was a model failure. Retrieval did exactly what retrieval is supposed to do, which is find the passages most relevant to the question. The model did exactly what a good model does, which is synthesize what it was handed into fluent, faithful prose. The security decision that produced the leak was made much earlier, by someone connecting the assistant to the document store, the CRM, and the wiki, at the moment they had to answer one small configuration question that almost nobody treats as a security architecture decision: as whom does this system read?

Every integration asks one quiet question, and most answer it wrong

There are only two real answers to that question, and they are wildly different in cost. The first is to give the AI system its own service account — a single machine identity, granted enough access to be useful across the whole corpus, reading every document on behalf of the platform rather than on behalf of any particular person. This takes an afternoon, works immediately, and demos beautifully, because a system that can see everything appears vastly more capable than one that cannot. The second answer is to make every retrieval happen under the identity of the human who asked, with delegated credentials, entitlements captured at index time and evaluated at query time. That approach is weeks of unglamorous work, it makes the demo look worse, and it is the only one of the two that is actually a security boundary.

What makes the first answer so dangerous is not that it is permissive in some general sense. It is that a service account's permissions are the union of what the organization's people can see, and a union is not an average — it is the ceiling. The most sensitive material in any enterprise sits under the narrowest access lists: the compensation model, the reorganization plan, the litigation folder, the board deck, the deal desk's floors. Those are precisely the documents a broad service account must be granted in order for someone, somewhere, to get useful answers about them, and once granted, that account carries them into every conversation it has with anyone. The AI system stops being a search tool and becomes something closer to a very fast, very fluent declassification pipeline, which is a strange thing to install in the middle of a company without meaning to.

The reason this evades the controls organizations already trust is that summarization strips exactly the signals that used to carry sensitivity. In the ordinary world, a person encountering a restricted document also encounters the folder it lives in, the header stamped across the first page, the sharing dialog that would refuse them, the awkward moment of requesting access from an owner who would ask why. An answer that arrives in a chat window carries none of that. The provenance has been flattened into a sentence, and a sentence has no permissions attached to it. Meanwhile the surveillance built for the old failure modes sees nothing worth flagging, because from the perspective of the document platform's audit log, a legitimate, well-behaved service account read a file it was fully entitled to read. No file was downloaded, no attachment left the tenant, no rule fired. The only trace is a helpful paragraph in someone's chat history.

Retrieval has to ask on behalf of the person, not on behalf of the platform

Fixing this is unglamorous and largely structural, and it happens almost entirely below the model. Entitlements have to become first-class metadata on every chunk that enters the index — not a note about which repository a document came from, but the resolved access list that governs it — and that metadata has to be kept alive as reality changes, because a document reshared on Tuesday and an employee who moved teams on Wednesday both create a silently stale index that is a latent breach waiting for the right question. When a query arrives, the asking user's identity has to be expanded into their full effective group membership, nested groups and inherited permissions and explicit denials included, and that expansion has to constrain the search itself rather than clean up after it. Filtering results after ranking feels equivalent and is not: post-hoc filtering leaks through result counts, through citation titles rendered before the body is suppressed, through the conspicuous difference between an answer that says nothing exists and one that declines to elaborate. The entitlement has to be part of the retrieval predicate, so that unauthorized passages are never candidates in the first place.

It is worth being blunt about the approach that gets reached for instead, because it is everywhere. Writing into the system prompt that the assistant must not disclose compensation data or unreleased financials is not an access control; it is a stated preference, expressed in the same text channel as everything else the model reads, evaluated by a probabilistic system that has no privileged way to distinguish an instruction from a document. Output classifiers that scan responses for sensitive content are a reasonable defense-in-depth layer and a poor primary boundary, for the simple reason that by the time they run, the restricted text has already been placed in the context window — and the context window is not an ephemeral thing. It is copied into request logs, distributed traces, evaluation datasets, prompt caches, and vendor-side retention, and those systems are almost always governed by weaker access rules than the documents whose contents they now contain. A control that lets sensitive data into the pipeline and then tries to catch it on the way out has already lost the argument about where the boundary lives.

Why this quietly decides whether the pilot ever ships

The commercial consequence of getting this wrong shows up later than the security consequence, and it is the reason so many promising internal AI programs die in a conference room rather than in production. Gartner has predicted that more than forty percent of agentic AI projects will be canceled by the end of 2027, naming escalating costs, unclear business value, and inadequate risk controls among the causes. The pattern that produces the third of those is depressingly consistent. A pilot succeeds precisely because it was wired with a permissive service account, since that is what made it feel magical to the executives who saw it. The production review then asks the entitlement question for the first time, discovers there is no answer, and finds that retrofitting per-user access into a corpus indexed without it is a larger project than the one that got approved. The program does not fail because the model underperformed. It fails because identity was treated as a deployment detail rather than as the architecture, and the bill came due at the worst possible moment.

This is also why the problem cannot sensibly be solved one application at a time. Every assistant, every agent, every new connection to a system of record re-poses the same question, and an organization that answers it individually will answer it inconsistently and eventually answer it wrong. Identity propagation belongs at the platform layer, where connections to the systems of record are made once and every workload above them inherits the same behavior — which is the argument behind treating enterprise knowledge as a governed layer rather than a pile of ingested files, and behind wiring Model Context Protocol connections and assistants so that they carry the asking user's entitlements into every tool call rather than the platform's. It is the design StudioX builds toward with Enterprise Knowledge and Enterprise Deployment: agents that reason over what a given person is entitled to reason over, with human-in-the-loop checkpoints on the actions that cross a boundary, and a gateway that can actually see and record which identity asked for what. The growing body of work on the autonomous enterprise keeps circling this same conclusion from different directions — that autonomy is bounded by identity, and a system permitted to act on behalf of everyone is not more autonomous, only less governable.

The mental model worth carrying away is a replacement for the one most teams are using. Stop asking what the AI system knows, which is an interesting question with no security content, and start asking who the system is when it reads. For any answer it produces, the test is whether the person who asked could have assembled every fact in that answer themselves, through the ordinary systems, with their own credentials and enough patience. If the answer is yes, the system has no blast radius of its own; it is an accelerator on access that already existed, and its risk profile is the risk profile of your existing permissions model. If the answer is no, then what has been deployed is not an assistant but an undocumented path around the access controls the organization spent years building — and as these systems move from answering questions to taking actions, that same question will have to be asked again about writes, where the wrong answer does not merely reveal something but changes it.