Automate agent evaluation using the Power Platform API.

Copilot Studio provides users with tools to continuously evaluate agent performance by running automated tests based on predefined test suites using the Power Platform REST API. Using the REST API, you can programmatically enable agent evaluation as part of the development process, such as during agent updates, release validation, or regression testing.

Automating the assessment process helps you:

• Verify agent quality after making changes.
• Run periodic performance tests on agents in production or staging environments.
• Integrate testing agents into CI/CD pipelines.
• Detect regression errors in agent behavior early in the development lifecycle.

Prerequisites

• You have the Bot ID and Environment ID for the target agent.
• A test suite is created in Copilot Studio for your target agent.
• A user access token issued by Microsoft Entra ID (OAuth 2.0).
  • Obtain the access token using the client ID of the application subscription with the appropriate scope provided by the Power Platform API.

Overview of running assessments using REST API

To run an assessment using the Power Platform API, follow these general steps:

1. Fulfill the prerequisites.
2. Find and retrieve the ID of the test suite you want to use.
3. Run the evaluation.
4. Retrieve results using the evaluation run ID.

When the request is successful, the evaluation process will run asynchronously and produce results that you can review in Copilot Studio.

API operations for automating assessments

Copilot Studio supports REST API operations that you can use to trigger programmatic evaluation of your agent using an existing test toolkit.

How to obtain the test agent toolkit

• Endpoint :GET https://api.powerplatform.com/copilotstudio/environments/{EnvironmentId}/bots/{BotId}/api/makerevaluation/testsets?api-version=2024-10-01
• Purpose : To retrieve an array of test kit IDs and other details for a specific agent.
• Response : Returns a list of valuetest toolkit names with the following information:
  
  • auditInfoTimestamps and user IDs are used to create and modify each test kit.
  • displayName: Name of the test kit.
  • id: ID of the test kit.
  • descriptionDescription of the testing toolset.
  • state: The state of the test suite. An available test suite has the state Active.
  • totalTestCasesThe number of test cases in the test toolkit.

Get details of the test agent toolkit.

• Endpoint :GET https://api.powerplatform.com/copilotstudio/environments/{EnvironmentId}/bots/{BotId}/api/makerevaluation/testsets/{TestSetId}?api-version=2024-10-01
• Purpose : To retrieve details for a specific test suite, using the test suite ID.
• Response : Returns information about an item in the response array.

Start evaluating agents

• Endpoint: POST https://api.powerplatform.com/copilotstudio/environments/{EnvironmentId}/bots/{BotId}/api/makerevaluation/testsets/{TestSetId}/run?api-version=2024-10-01
• Purpose : To run an evaluation for a test suite using its test suite ID. You can also include a user profile to authenticate the connection during the evaluation run. Use this mcsConnectionIdto specify the user profile. If you don't add it mcsConnectionIdto your call, the evaluation will run without authentication.
• Response : Returns the following information:
  • runId: The ID of the evaluation run. Use this ID to retrieve evaluation details.
  • lastUpdatedAt: The time the status was last updated.
  • executionState: The status of the run while the evaluation process is underway.
  • state: Current state of the run.
  • totalTestCases: The total number of test cases in the test suite used for evaluation.
  • testCasesProcessed: Total number of test cases evaluated up to the last update.

Retrieve information on the agent's test runs.

• Endpoint :GET https://api.powerplatform.com/copilotstudio/environments/{EnvironmentId}/bots/{BotId}/api/makerevaluation/testruns?api-version=2024-10-01
• Purpose : To retrieve an array of all previous runs.
• Response : Each item in the array contains the same values ​​as those found in Get agent test run details.

Get details about the agent's test run.

• Endpoint :GET https://api.powerplatform.com/copilotstudio/environments/{EnvironmentId}/bots/{BotId}/api/makerevaluation/testruns/{TestRunId}?api-version=2024-10-01
• Purpose : To retrieve detailed information from a single assessment for use runIdin your target assessment run.
• Response : Returns the following information:
  
  • id: The ID of the evaluation run. Use this ID to retrieve evaluation details.
  • environmentId: Your agent environment ID.
  • cdsBotId: Target agent ID.
  • ownerId: The user ID has started the evaluation process.
  • testSetId: ID of the test kit used for evaluation.
  • state: Status of the evaluation process.
  • startTime: Time when the assessment begins.
  • endTime: The end date of the evaluation, if applicable.
  • name: Name of the evaluation process.
  • totalTestCases: Total number of test cases in the test toolkit.
  • mcsConnectionId: The Copilot Studio connection ID of the user profile used for the assessment process. Returns nullif no user profile is connected.
  • testCasesResults: A list of test cases during the evaluation process. Includes:
    • testCaseId: Test case ID.
    • state: The test case is complete.
    • metricsResultsDetails and results for each test method used for the test case. Including:
      • typeTesting methods.
      • resultThe final test results for this test case. These include:
        • dataDetails of the results. The exact values ​​depend on the testing method. Learn more in the Power Platform API documentation. For a general quality test, the feedback includes the following:
          • abstentionDid the agent answer the question?
          • relevanceIs the answer relevant or not?
          • completenessIs the answer complete or not?
      • status: The state of the test case.
      • errorReasonIf an error occurs, what is the cause of the error?
      • aiResultReasonAI's explanation of the test case results.

Use Microsoft Studio Connector ID for evaluation.

To begin evaluating the agent, you can optionally add your Microsoft Studio Connector ID to the call as a user profile for the evaluation process. To find mcsConnectionIdyours:

1. Access Power Automate .
2. Open the Connections page .
3. Select Microsoft Copilot Studio connection .
4. Copy mcsConnectionIdfrom URL: ../connections/shared_microsoftcopilotstudio/{mcsConnectionId}/details

You've just finished reading the article "Automate agent evaluation using the Power Platform API." edited by the TipsMake team. We hope this article has provided you with many useful tech tips and tricks. You can search for similar articles on tips and guides. Thank you for reading and for following us regularly.