Add public websites as knowledge sources for agents in Copilot Studio.

By adding publicly available websites as knowledge sources at the agent level, you are using publicly available information to expand the agent's reach.

Add another public website

To add a public website as a knowledge source, follow these steps:

1. Open agent.
2. Select Add knowledge from the Overview or Knowledge page , or from the Properties of the Generative Answers node.
3. Select Public websites .
4. Enter the URL of the public website. To use a variable instead of a static URL, select the variable selector icon next to the URL field and choose Custom , System , or Environment .
5. Add a name (by default, the website URL will be used).
6. Add a description. Provide a detailed description, especially if Generative AI  is enabled, as the description aids in orchestrating generative processes.
7. Select Add to agent  to complete adding the knowledge source.
8. Test your agent to see how well it responds to content-related questions from the website. You may want to test exception questions to decide if you need a lower level of moderation for greater coverage.

Factors to consider regarding URLs

The URLs you use in your agent represent the scope of content for generating responses. Some URLs have requirements and restrictions.

Note :

1. When you specify a URL, the knowledge source will use Grounding with Bing Search to return information from the web.
2. The Classic data options in the node's Generative Answers property allow you to specify which URLs are dynamically generated using PowerFx formulas. 
3. You can also specify Bing Custom Search in the Classic data options of the Generative Answers node property. 

URL types and structures

1. URLs can have up to two levels of depth, with subpaths indicated by forward slashes (/). However, a trailing slash is still permitted.

Valid	Invalid
www.contoso.com www.fabrikam.com/engines/rotary www.fabrikam.com/engines/rotary/	www.fabrikam.com/engines/rotary/dual-shaft

1. If the URL redirects to a different top-level website, that content will not be included in the results:

For example, if www.fabrikam.comyou redirect to www.contoso.fabrikam.com, your agent will not generate a response from the content on either of those URLs.

1. URLs that point to websites requiring authentication or those not indexed by Bing will not be supported.

For example, wikis and SharePoint sites require authentication, so you can't use them:

1. fabrikam.visualstudio.com/project/_wiki
2. fabrikam.sharepoint.com

Domain URL structure

Your agent uses any publicly viewable content within the URL you specify, including subdomains under the top-level domain, to generate content.

For example:

1. If you use it www.fabrikam.com/engines/rotary, the agent will also use the content above www.fabrikam.com/engines/rotary/dual-shaftto generate a response.
2. Agent does not use the content above www.fabrikam.com/tools, because tools is not a subdomain of rotary .
3. If you use the content above www.fabrikam.com(with www), the agent will not use the content above news.fabrikam.com(without www), because news.is a subdomain of the top-level domain fabrikam.com.
4. If you use fabrikam.com , the agent will use content from both www.fabrikam.comnews.fabrikam.com , as they both reside under the top-level domain fabrikam.com.

Social media and forum URLs

The agent may generate meaningless, irrelevant, or inappropriate responses if you use a forum or social media URL. Therefore, community content on social media often increases the risk of more rejected responses.

Artificial intelligence is trained to avoid generating toxic and offensive responses.

Search engine URL

Do not include URLs of search engines like bing.com, as they do not provide helpful feedback.

Use variables as URLs.

Use variables in the URL of a public website or SharePoint knowledge source to dynamically control the scope of content your agent uses.

Instead of creating multiple knowledge sources for different scenarios, define a single knowledge source and insert a variable into the URL. During runtime, Copilot Studio will resolve the variable and use the resulting URL for location.

Variable support

When adding or editing the URL of a public website or SharePoint knowledge source, you can insert a variable into the URL field.

Use variables to:

1. Add the path to the base URL. https://www.contoso.com/{variable}
2. Replace the entire URL  {variable} 

At runtime, the variable will be resolved to a string value. The knowledge source is scoped according to the resolved URL.

Note : Using the variable will change the URL used for locating. It does not change the URL structure requirements or depth limits for public web pages.

Create and set variables

To use variables in a public website or SharePoint knowledge source:

1. Create a string variable with global scope .
2. Set the variable value using one of the following options:
   1. Subject input
   2. User profile attributes, such as language.
   3. Environmental variables
   4. Agent flow or connector
3. Insert a variable into the URL field by selecting {x} (Insert variable) .
4. Check the agent to confirm the variable is resolved to the desired URL.

Example: Product range

You can limit the scope of knowledge to different product areas based on user queries.

1. Use the subject input to collect the product name as a variable named Product.

2. Add a Set variable value node after the Trigger node and create a new variable named ProductTable.

3. Use the Power Fx formula to determine and map the product name to the URL.

Table( { Product: "Copilot", Path: "microsoft-365-copilot" }, { Product: "Teams", Path: "microsoft-teams" }, { Product: "Surface", Path: "surface" } )

4. Add another Set variable value node and use it to create a new system-wide string variable named GlobalProductURL.

5. In the To value settings , select Formula and use the Power Fx function to perform a lookup in the table based on the value of the input variable Product, Topic.Product, as follows:

LookUp(Topic.ProductTable, Product = Topic.Product, Path)

6. Using variables in knowledge source URLs:

https://www.microsoft.com/en-us/{GlobalProductURL}

The agent only queries the limited scope of the webpage at runtime.

Common cases

Use variables in public website knowledge sources for:

1. Product-based routing: Restrict queries to specific product documents.
2. Filter by region or language: Map User.Languageto localized URLs (e.g. en-us, it-it).
3. Environment-specific URLs: Switch between development, testing, and production websites.

Notes

1. Knowledge sources on public websites use Bing search and rely on indexed content.
2. URL depth limits still apply when you use variables.
3. Variables only change the destination URL. They do not affect how the content is indexed or retrieved.

Important note : The variable must resolve to a valid public URL that meets the requirements for knowledge sources on a public website. Invalid or unsupported URLs may not return results.