Home / Insights

Testing Copilot Studio agents: how to know it actually works

A few promising test questions and some good answers feel like evidence. They are not. The people who use your agent will phrase questions differently, hold longer conversations, ask things you never planned for and have different permissions from the person who built it. The first week in production is a poor time to find that out.

, 8 minute read

Why a few good answers are not evidence

Agents do not behave the same way twice. Their answers shift with changes to the model, the knowledge, the instructions and the tools. That is why they need a different kind of testing from the software most teams are used to. This article sets out a practical approach: build a real test set, choose measures that fit, test as different users, and keep measuring after go-live. Evaluation features in Copilot Studio are still evolving, with some in preview, so check what is currently available in your environment.

Why AI agents need different testing from other software

Traditional software testing checks that a defined input produces a defined output. If it passes today, it will pass tomorrow unless someone changes the code. Agents break that assumption. The same question can produce differently worded answers, a change to a knowledge document can alter results without anyone touching the agent, and a model update can change behaviour overnight.

So the goal of testing shifts. You are not proving that the agent is correct once. You are building a repeatable way to see how it behaves, spot when that behaviour drifts, and judge whether a change made things better or worse. Agent evaluations exist for exactly this. They make variability visible and manageable, so you can launch an agent knowing how it behaves rather than hoping.

Manual testing still has a place. Chatting with the agent in the test pane is the quickest way to explore and to find obvious faults. It just cannot answer the question “did my last change make things worse?” across fifty scenarios.

Build the test set from real questions

A test set is a collection of scenarios with, where possible, a description of a good answer. Its quality decides the value of everything that follows, and the best source is real user language. Mine service desk tickets, shared mailbox messages, intranet search logs and the questions people ask in team chats. Interview a few of the intended users and ask what they would want to know.

A set of a few dozen well-chosen scenarios is enough to start. Aim for a deliberate mix:

  • Core questions. The ten or twenty things the agent exists to answer, phrased the way people really phrase them.
  • Variations. The same question asked formally, casually, with typos and with missing context.
  • Multi-step conversations. Follow-up questions that depend on earlier answers.
  • Out-of-scope questions. Things the agent should decline or hand over, such as legal advice or individual pay.
  • Awkward and adversarial cases. Questions that try to push the agent beyond its instructions, or that include instructions hidden in the text.
  • Questions with no answer in the knowledge. The agent should say so rather than invent one.

Write the expected answer or the key facts it must contain. Ask the subject-matter owner to check them, because an agent can only be judged against a standard that someone agreed to. Copilot Studio lets you create test sets manually, generate them with AI assistance or import them from a spreadsheet, so a first set does not take long to assemble.

Choose measures that fit the question

Different questions need different ways of scoring an answer. Copilot Studio’s evaluation features offer several test methods, and the set available varies by experience and is changing. The principle is to match the method to what you are checking:

What you are checkingType of measure
A fixed fact, such as a form name or a policy number, must be exactExact match or keyword match
The answer should mean the same as an approved answer, though the wording can differCompare meaning or text similarity
There is no single right answer, but quality still mattersGeneral quality, scored by a language model against criteria such as relevance and completeness
The agent should call the right tool, or not call oneTool use check
Answers must not contain harmful contentContent safety check
You have a rule specific to your organisationCustom criteria with your own labels and instructions

Two cautions apply. AI-scored measures are useful and scalable, but they are not a substitute for human review, so read a sample of results yourself, especially the failures and the borderline passes. And do not rely on one measure. A response can be well written and still be wrong, or exact and still unhelpful.

Test as different people, and test the tools

Two kinds of testing are easy to forget. The first is permissions. If your agent uses SharePoint or other sources that require sign-in, it only returns what the signed-in user can see. That means an answer that looks perfect when the builder tests it may be empty, or different, for an ordinary employee. Test with accounts that represent real groups, including someone with limited access and, where it matters, someone who should be refused. Evaluations and tool connections run under the profile of the person running them, so plan which account you test from.

The second is tools. If the agent calls a flow or connector, test more than the success case. Does it call the right tool for the request? What happens when the person gives incomplete information? What does the agent say when the system it depends on is slow or unavailable? Does it avoid calling a tool when it should simply answer? A tool that works perfectly in isolation can still be used badly by an agent, and the description you gave it is usually the cause.

Make regression testing a habit

Run the full test set every time something changes: the instructions, a knowledge source, a tool or the underlying model. Compare the results with the previous run. You are looking for two things: whether the change fixed what you intended, and whether it quietly broke something else. The second is the one you would otherwise miss.

Change one thing at a time where you can. If you rewrite the instructions, add two knowledge sources and change a tool description in one go, a drop in quality leaves you guessing. Small, deliberate changes turn evaluation into a quick feedback loop instead of a monthly ceremony.

Remember that testing has a cost. Running evaluations uses Copilot Credits, and Copilot Studio shows estimated usage as you set up a run. That is a reason to keep your test set focused and to run the full suite when it matters, not a reason to skip testing.

Keep watching after go-live

Testing before launch tells you about the questions you predicted. Analytics after launch tells you about the ones you did not. Copilot Studio’s analytics show conversation volume, how topics perform, how often the agent resolves a conversation and user satisfaction signals. Those numbers are most useful when you read them alongside the conversations themselves.

Set aside regular time, weekly at first, to read a sample of real transcripts. Look for questions the agent could not answer, answers people rejected or rephrased, conversations that ended in escalation and questions that revealed a gap in your content. Each one is either a fix to make or a new test case to add. Over time your test set grows from a guess about what people ask into a record of what they actually ask.

Agree thresholds in advance, and who decides. For example, what proportion of test scenarios must pass before a change goes live, and what level of unresolved or escalated conversations triggers a review? Without an agreed line, every decision becomes a debate.

A simple improvement loop

When a test fails or a transcript shows a poor answer, resist the urge to fiddle with the instructions. Work out the cause first. Most failures fall into a small number of buckets, and each has a different fix:

  • The content is missing or outdated. Fix the source document, not the agent.
  • The content exists but was not found. Check that the right source is connected, that the person has access, and that the document is clearly written.
  • The instructions are unclear. Tighten the scope, add an example, or state what the agent should do when it is unsure.
  • The wrong tool was used. Rewrite the tool description and its inputs.
  • The question is out of scope. Improve the decline and handover message.

Fix one thing, rerun the test set, and confirm the result before moving to the next. It is a slow-looking method that is faster than the alternative, which is a growing list of changes nobody can explain.

Questions for your next agent review

  • Do we have a test set built from real questions, and who approved the expected answers?
  • When did we last run it, and what changed since?
  • Have we tested as users with different access, not just as the builder?
  • Do we know how the agent behaves when a tool fails or the knowledge has no answer?
  • Who reads the transcripts, how often, and what happens to what they find?
  • What result would make us pause or roll back a change?

An agent you cannot measure is an agent you cannot improve. Teams that build this discipline early find that they can expand what their agents do with confidence, because they can see the effect of every change. Those that skip it often find that trust, once lost, is hard to rebuild.

Flex helps New Zealand organisations get Microsoft Copilot and Copilot Studio working on real processes. If you would like to talk through your own rollout, get in touch with the Flex team.

Related reading

Related Flex services

Sources and further reading

Questions we are asked

How do you test a Copilot Studio agent?

Build a test set from real user questions, including awkward ones, decide what a good answer looks like for each, run the set as different types of user, and re-run it whenever the agent, its knowledge or the model changes.

Why do agents need regression testing?

Because their answers shift when instructions, knowledge sources, tools or the underlying model change. Re-running the same test set after every change shows whether a fix in one place has caused a problem somewhere else.

What should we monitor after go-live?

Read real conversations regularly, track questions the agent could not answer, handovers to people and user feedback, and feed what you find back into the test set.

Talk to us about what you would automate first

Your first consultation is free and comes with no obligation. It is a working session, not a pitch: bring a process that is costing your team time and we will tell you plainly whether automation fits it, which platform we would use, and what a pilot would look like.

Book a free consultation