How to Choose Copilot Studio Agent Evaluation Methods

Copilot Studio evaluation methods answer different questions. General quality judges whether a response is useful and grounded; Compare meaning checks the intended answer; Tool use verifies orchestration; keyword and exact-match methods check deterministic content; and Custom applies criteria you define. Use the smallest combination that proves the behavior you care about.

Evaluation methods compared

Method	Best use	Test-set support	Required case data
General quality	Open-ended helpfulness and grounding	Single response and conversation	None
Compare meaning	Answers that may be phrased differently	Single response	Expected answer and pass score
Tool use	Required tool or topic invocation	Single response	Expected capabilities
Keyword match	Required terms or disclosures	Single response and conversation	Keywords; Any or All
Text similarity	Wording that should resemble a reference	Single response	Expected answer and pass score
Exact match	Fixed codes, numbers, or strings	Single response	Exact expected answer
Custom	Policy, style, or domain-specific rubric	Single response and conversation	Instructions and labeled outcomes

Add a method to a test set

Create or edit a test set and select Add test method.
Select one or more methods, then choose OK.
Set a pass score where required.
Open each test case and add the expected answer, keywords, or capabilities required by that method.
Select Save.

A case with missing required data can produce an Invalid result rather than a meaningful pass or fail. Validate the test-set configuration before comparing runs.

General quality

General quality uses a model-based evaluator to consider relevance, groundedness, completeness, and whether the agent appropriately answered or abstained. It is the best starting point for open-ended responses where several answers could be acceptable and no fixed reference is available.

Because another model performs the assessment, review important failures and passes manually. A high score does not prove factual or policy compliance, and evaluator behavior may not match a subject-matter expert in edge cases.

Compare meaning

Use Compare meaning when the answer must communicate certain facts or intent but can use different wording. Add a concise, correct reference response and set a passing threshold. This is often more appropriate than exact match for support answers, explanations, and summaries.

Do not pack several optional details into the reference if their absence should not cause failure. The expected answer should represent the minimum correct meaning.

Tool use

Tool use checks whether the agent invoked an expected tool or topic. Use it when orchestration is part of the requirement—for example, an order-status question must call the order system rather than invent a result from general knowledge.

A tool call alone does not prove that the final answer is correct. Pair Tool use with an answer-quality or deterministic check when both execution and response matter.

Keyword match

Keyword match passes when any or all configured phrases appear. It is useful for mandatory disclosures, product names, escalation instructions, or required terminology.

Keep phrases specific enough to be meaningful. A response can contain the keyword and still contradict the policy, so use this as a narrow assertion rather than a complete quality measure.

Text similarity

Text similarity scores how closely the agent's answer resembles the expected response in wording and meaning. It can help when format and phrasing should remain relatively stable but do not need to be character-for-character identical.

If meaning matters more than wording, Compare meaning is usually clearer. Similarity can penalize valid concise answers or reward verbose answers that copy reference language without satisfying the user.

Exact match

Exact match compares the full response character by character. Use it only for deterministic output such as a code, fixed phrase, or strictly formatted value. Whitespace, punctuation, or explanatory text can cause failure.

Do not use exact match for natural-language answers. It turns harmless paraphrasing into noise and encourages brittle agents that optimize for a test string rather than the user.

Custom evaluation

A Custom method uses your instructions and labels to classify the response. It is suitable for a domain rubric such as:

Does the HR answer avoid requesting unnecessary sensitive data?
Does the support answer state the correct escalation condition?
Does the response follow an approved tone and include the required limitation?

Write instructions that cite observable features of the response. Define at least two labels with distinct descriptions, and assign each label a Pass or Fail outcome. Avoid vague criteria such as “good” or “professional” without explaining what those terms mean.

Practical method combinations

Knowledge assistant: General quality + Compare meaning for critical facts.
Transactional agent: Tool use + Compare meaning or Exact match for the returned value.
Regulated response: Custom policy rubric + Keyword match for a mandatory disclosure.
Fixed command output: Exact match alone may be sufficient.
Multi-turn support flow: General quality + Custom checks for required clarification and safe escalation.

Avoid misleading scores

Do not add many overlapping methods merely to create more metrics.
Review pass thresholds against known good and bad examples.
Keep a human-approved reference set for consequential workflows.
Investigate Invalid and Error results separately from genuine failures.
Use the same cases, profiles, connections, and methods when comparing agent versions.

Microsoft's evaluation-method documentation lists the current support and configuration for each method. See our instructions for editing Copilot Studio test cases and updating a complete test set.