Work the order of operations, because the model is last
When someone tells me Copilot gives them rubbish answers, the model is the last thing I look at and it is almost never what I end up changing. The causes sit in a predictable order, and going through them in that order takes about an hour rather than the fortnight people usually spend arguing about it.
The order I use:
- Which product were they actually using.
- What can it see, and what has been deliberately hidden from it.
- Is the content it found any good.
- Is the question one this tool should be answering at all.
Only after those do I care about phrasing, and by then the phrasing usually fixes itself.
Half the complaints are about a different product
Copilot is a family name, and users do not know which member they met. Copilot Chat is included at no extra cost with eligible Microsoft 365, Office 365 and Teams licences, which is a different thing from the Microsoft 365 Copilot add-on at $30.00 per user per month paid yearly. A person who has been told the organisation “has Copilot”, and who is comparing what they see with what a colleague in another team sees, will conclude the product is inconsistent and unreliable. They are describing a licensing fact, not a quality problem.
Two related checks. On-premises Office is not eligible for Copilot, so a user still working in an on-premises client is not going to get the experience the demo showed them. And if the complaint is about an agent rather than chat, ask which runtime it was built on, because Copilot Studio now exposes three harnesses: the GitHub Copilot harness for reasoning-heavy multi-step work that natively creates and edits Word, Excel, PowerPoint and PDF; the standard harness for rule-based agents and agent flows; and the Copilot chat harness for extending Copilot Chat with enterprise knowledge. An agent that gives shallow answers to a multi-step question may be a correctly built agent on the wrong runtime.
Talking about “Copilot” as one product is now the single most common source of confusion in these conversations, including in my own writing until recently.
Then find out what it can and cannot see
The tool retrieves what the user is already permitted to open. That cuts both ways, and both hurt.
One direction is oversharing: it surfaces things the user could technically always reach and never would have found. That is a permissions problem wearing a Copilot costume, and it is not the subject of this piece.
The other direction is the quiet one. Somebody applied a control and nobody told the users. Restricted Content Discovery is a site-level setting in SharePoint Advanced Management. It does not change permissions and it does not remove content from the index, but it does remove the AI entry points from the site: the Copilot button, the AI actions menus including agent creation, and “Create pages with AI”. SharePoint sites only, not OneDrive. Microsoft’s own documentation cautions that excessive use degrades the quality of Copilot answers, which is not marketing softness, it is a straight description of what happens when you hide the good content from the retriever.
So when a team says answers about their own area are thin, find out whether a governance decision made six months ago is the reason. Often it was the right decision at the time, taken under time pressure, and nobody has revisited it since the content got tidied up.
Then look at what it found, because it is probably a mess
If two documents in your tenant contradict each other, the answer will contradict itself, and the model is being obedient rather than stupid. This is where most of the remaining complaints live: three versions of the policy, a 2023 process note that nobody archived, a rate card in a personal OneDrive, a handbook whose owner left. Ask “which of these is current” and the honest organisational answer is frequently “ask Jan”, which is not a fact the retrieval layer has access to.
The fix is unfashionable and it is content ownership. Not a metadata programme. A named owner for the twenty documents that matter most in each function, a review date, and the deletion of the superseded copies rather than a helpful note at the top saying they are superseded. Do that for one function and its answers improve within a week, which is the demonstration that makes the second function volunteer.
Last, ask whether the question was fair
Some questions cannot be answered from what is in the tenant. Anything living in a line-of-business system the tool has no route to. Anything that depends on a decision taken verbally. Anything where the true answer is a judgement somebody has to own. Users ask these questions because we told them the tool understands their work, and then we blame their prompting when it does not.
Doing this in order also changes what you do next. If the causes were licensing and a governance setting, no amount of training would have touched it. That is worth knowing before you commission a training programme to fix an information architecture problem, which I have watched happen more than once and have been paid to help with, which is not my finest professional memory.