Microsoft 365 Copilot is a permission system, not just GPT-4 in Office

Microsoft introduced Microsoft 365 Copilot on March 16, 2023 as an AI layer for Word, Excel, PowerPoint, Outlook and Teams. The easy headline was that GPT-4 had arrived in Office. The announced architecture said something more important: the model would be combined with Microsoft Graph, applications and authorized work data. That changes the procurement question. A company must assess not only whether the model writes well, but which information it retrieves, under whose identity, what it can do and who is accountable when a persuasive output is false.

The product is not the model

In its original announcement, Microsoft described an orchestration engine joining large language models, including GPT-4, with Microsoft 365 and Graph. Business Chat could use calendars, email, chats, documents, meetings and contacts to prepare contextual answers. Inside applications, Copilot promised to draft in Word, analyze in Excel, create presentations, summarize threads and capture decisions.

That is at least five components: model, information retriever, permission system, interface and receiving application. Failure can begin in any one. The model can fabricate; retrieval can select an old version; inherited access can expose a forgotten shared file; the interface can hide provenance; the user can accept without checking. Measuring only GPT-4 accuracy does not measure a product operating on corporate data.

“Grounded in your data” cuts both ways

Connecting a model to internal work addresses one limitation: GPT-4 cannot inherently know the latest proposal, this morning’s meeting or numbers in a private sheet. Retrieving relevant material can ground an answer. The same access expands the impact of failure. A summary can mix projects, a deck can include an obsolete figure, and an email can reveal a correct fact to the wrong recipient.

Microsoft said Copilot models would not train on tenant data or prompts and would show only information the user could access. These are meaningful commitments, but not a substitute for internal audit. “The employee can access it” describes an inherited maximum, not evidence that every permission remains necessary. A folder shared years ago or an overbroad group can be technically accessible yet inappropriate for instant synthesis.

Map information before the pilot

An organization needs to know which repositories are included, who owns them, which sensitivity classes they contain and how long they retain material. It should then review groups, shared links and exceptions. The goal is not to clean the entire corporate archive before trying AI. It is to define an initial zone where permissions and document quality are understandable.

A good pilot begins with one process and one collection, not every employee and every document. It might summarize approved minutes inside one team. Legal drafts, personnel files, trade secrets and privileged communications remain outside. This negative list must be written before the demonstration, while enthusiasm has not yet turned every boundary into an inconvenience.

The source must travel with the answer

Microsoft said Copilot would note limitations, link sources and prompt users to review and fact-check. That is essential. A useful enterprise answer should open the exact document and show its date, author and supporting passage. A filename without a version is insufficient. So is a bundle of links if readers cannot tell which source supports each claim.

Traceability enables three checks. Provenance asks where a fact came from. Currency asks whether it is the valid version for this decision. Authority asks whether the document could set policy or was an informal note. A model can summarize the wrong contextual source perfectly. Review must ask not only whether it paraphrased correctly, but whether it chose a document capable of answering.

Each application needs a different threshold

A Word draft is reversible and can be reviewed before circulation. An Excel analysis may feed a budget; a bad formula or filter carries greater consequences. A Teams summary can attribute an agreement nobody made. An Outlook message can leave the organization with one click. Calling all of this productivity assistance hides differences in severity and reversibility.

Classify functions in three levels. First, generate or summarize without publishing: a person always validates. Second, modify shared content: require source comparison and logging. Third, communicate externally, change access or trigger a process: require a separate visible approval.

Human review needs a contract

“The user is in control” does not specify what must be checked or how much time is available. If Copilot produces ten times more text, a person can become a rubber stamp. Each workflow needs a review contract: critical fields, sources to open, error tolerance, authorized role and approval record. The system should abstain or escalate when evidence is missing, not cover the gap with fluent language.

OpenAI’s GPT-4 introduction said the model remained far from perfect on factuality, steerability and guardrails. Adding context does not erase those limits. It can reduce unsupported answers when retrieval finds the right document, but it can also supply private material for a more specific and credible error. Review must be designed around possible harm, not average model performance.

Evaluate the complete system

Tests should use real tasks with known answers. Record whether the system retrieved the right source, respected access, distinguished versions, cited the passage, produced a factual result and abstained when information was insufficient. Adversarial tests should include embedded instructions, similar names, revoked files, conflicting figures and external recipients. One high-severity access failure can disappear inside an overall average.

The NIST AI Risk Management Framework, published in January 2023, supplies a useful discipline: govern, map context, measure and manage across the lifecycle. For Copilot, that means assigning an owner, describing affected people and harm, testing before expansion, monitoring incidents and having a withdrawal path. Evaluation changes whenever repositories, functions or user groups change.

Turn principles into controls

In 2022, Microsoft had published material supporting its Responsible AI Standard. The guide links transparency and reliability to interface design, failure planning, disaggregated evaluation, data documentation and human oversight. Buyers should ask for evidence that principles appear as product controls, not merely on a corporate page.

A minimum adoption file includes data architecture, effective permissions, retention policy, evaluation set, source logs, incidents, owners and suspension criteria. It should distinguish vendor statements from customer checks. When a property cannot yet be verified, label it as a limitation and narrow scope rather than filling the gap with confidence.

Productivity begins by reducing uncertainty

Saving drafting time is valuable, but a longer review, leak or decision based on an obsolete version can erase the benefit. Measure time to approved result, not time to first draft. Track corrections, opened sources, escalations, incidents and tasks that should not have been automated. Control is also productivity when it prevents rework.

The transferable skill is to assess any office copilot through six contracts: identity, data, retrieval, permissions, action and review. Each needs an owner, a test and a failure response. Microsoft 365 Copilot showed that value did not come simply from placing GPT-4 beside a document, but from connecting it to the right context without turning old access and persuasive text into automatic decisions.