How to Automate Copilot Studio Agent Evaluations With the Power Platform API

The Power Platform REST API can run an existing Microsoft Copilot Studio test set programmatically. A typical automation lists test sets, starts a run, records the returned run ID, polls until completion, and then evaluates the case-level results. Use this for regression testing and release checks, but keep human review for consequential agent behavior.

Prerequisites

• The target Copilot Studio Environment ID and Bot ID.
• A saved test set for that agent and its Test Set ID.
• A Microsoft Entra app registration with the appropriate Power Platform API delegated scope.
• A user access token for https://api.powerplatform.com.
• Permission for that user to access the environment, agent, test set, knowledge, and required connections.

Power Platform API currently uses delegated permissions for this workflow, so a headless service-principal design may not be supported in the same way as an interactive user flow. Follow the current Microsoft authentication documentation instead of copying a token from a browser. Never store bearer tokens or client secrets in source control or build logs.

API workflow

1. List the agent's test sets and select the intended ID.
2. Optionally retrieve the test-set details and validate its name and case count.
3. POST to the run endpoint with the run name and required connection settings.
4. Save the returned runId and callbackUri.
5. Poll with backoff until the state is Completed, Failed, Cancelled, or Abandoned.
6. Retrieve the completed run and inspect every case and metric.
7. Fail or warn the pipeline according to a version-controlled release policy.

Request headers

Authorization: Bearer <access-token>
Accept: application/json
Content-Type: application/json

Use the API version documented for the operation. The evaluation endpoints below currently use 2024-10-01.

List test sets

GET https://api.powerplatform.com/copilotstudio/environments/{EnvironmentId}/bots/{BotId}/api/makerevaluation/testsets?api-version=2024-10-01

The response contains a value array with fields such as id, displayName, description, state, totalTestCases, and audit information. Select by a configured ID rather than relying only on a display name that a maker can change.

Get one test set

GET https://api.powerplatform.com/copilotstudio/environments/{EnvironmentId}/bots/{BotId}/api/makerevaluation/testsets/{TestSetId}?api-version=2024-10-01

Use this check to confirm the expected test set exists and has the expected number of cases before triggering a release evaluation.

Start the evaluation

POST https://api.powerplatform.com/copilotstudio/environments/{EnvironmentId}/bots/{BotId}/api/makerevaluation/testsets/{TestSetId}/run?api-version=2024-10-01

A current request body can include:

{
  "evaluationRunName": "release-2026.08.11-commit-abc123",
  "mcsConnectionId": "<connection-id-or-empty>",
  "runOnPublishedBot": false,
  "toolsConnections": []
}

• evaluationRunName should identify the build, agent version, or release candidate.
• mcsConnectionId identifies the Microsoft Copilot Studio connection used for an authenticated run; leave it empty only when an anonymous run is appropriate.
• runOnPublishedBot selects the published agent when true and the draft when false.
• toolsConnections maps tool connection references to user connections when the evaluation needs them.

Do not reuse a highly privileged connection simply to make the evaluation pass. Use a dedicated account that represents the intended user and has only the necessary access.

Handle the asynchronous response

A successful start returns a status object rather than completed results. Record at least:

• runId
• callbackUri
• state
• executionState
• totalTestCases
• testCasesProcessed
• lastUpdatedAt

Poll the callback URI returned by Microsoft when available. Otherwise, query the run endpoint below. Use exponential backoff with a reasonable maximum interval and a total timeout. Treat Failed, Cancelled, Abandoned, and Unknown as explicit outcomes rather than looping forever.

List or retrieve runs

GET https://api.powerplatform.com/copilotstudio/environments/{EnvironmentId}/bots/{BotId}/api/makerevaluation/testruns?api-version=2024-10-01

GET https://api.powerplatform.com/copilotstudio/environments/{EnvironmentId}/bots/{BotId}/api/makerevaluation/testruns/{TestRunId}?api-version=2024-10-01

The detailed result can include run metadata, state, start and end times, connection information, and testCasesResults. Each case can contain metric results, the pass/fail state, evaluator data, error information, and the AI evaluator's reason.

Decide when a pipeline should fail

Do not gate a release on the aggregate pass rate alone. A useful policy might require:

• the run state to be Completed;
• no Error or Invalid result in mandatory cases;
• all security, permission, and compliance cases to pass;
• no regression in a protected group of critical cases;
• an overall threshold only after those conditions are satisfied.

Store the policy in version control and identify each critical case with a stable ID or tag. A transient service error should normally trigger a bounded retry and then an infrastructure failure, not be misreported as an agent-quality regression.

Connection ID and tool connections

An authenticated run may require the Copilot Studio connector connection ID and separate tool connection mappings. Microsoft documents connection details through the Power Platform connection resource. Resolve these identifiers at deployment time and keep secrets or tokens in the CI system's secret store.

Before automation, run the test set manually with the intended account to confirm that knowledge sources and tools connect successfully. API automation does not repair an invalid consent, expired connection, or missing permission.

Operational safeguards

• Use staging agents and non-production data for routine CI runs where possible.
• Limit concurrent runs; Copilot Studio may allow only one evaluation at a time for the agent.
• Log IDs, timestamps, states, and summary counts, but redact tokens and sensitive responses.
• Export or store required results because in-product retention is limited.
• Review API-version and schema changes before upgrading the integration.

Microsoft's evaluation automation guide, run-operation reference, and Power Platform authentication guide are the authoritative references. For manual diagnosis, see how to run and analyze Copilot Studio evaluations.