Skip to content

Commit a34945e

Browse files
howieleunghowieleung
committed
Update doc
1 parent d403ad4 commit a34945e

File tree

2 files changed

+52
-40
lines changed

2 files changed

+52
-40
lines changed

‎README.md‎

Lines changed: 52 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ The agent leverages the Azure AI Agent service and utilizes file search for know
44

55
<div style="text-align:center;">
66

7-
[**SOLUTION OVERVIEW**](#solution-overview) \| [**GETTING STARTED**](#getting-started) \| [**OTHER FEATURES**](#other-features) \| [**RESOURCE CLEAN-UP**](#resource-clean-up) \| [**GUIDANCE**](#guidance) \| [**TROUBLESHOOTING**](#troubleshooting)
7+
[**SOLUTION OVERVIEW**](#solution-overview) \| [**GETTING STARTED**](#getting-started) \| [**AGENT EVALUATION**](#agent-evaluation) \| [**OTHER FEATURES**](#other-features) \| [**RESOURCE CLEAN-UP**](#resource-clean-up) \| [**GUIDANCE**](#guidance) \| [**TROUBLESHOOTING**](#troubleshooting)
88

99
</div>
1010

@@ -57,10 +57,60 @@ Here is a screenshot showing the chatting web application with requests and resp
5757
| [![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/Azure-Samples/get-started-with-ai-agents) | [![Open in Dev Containers](https://img.shields.io/static/v1?style=for-the-badge&label=Dev%20Containers&message=Open&color=blue&logo=visualstudiocode)](https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/Azure-Samples/get-started-with-ai-agents) |
5858
|---|---|
5959

60-
Github Codespaces and Dev Containers both allow you to download and deploy the code for development. You can also continue with local development. Once you have selected your environment, [click here to launch the development and deployment guide](./docs/deployment.md)
60+
Github Codespaces and Dev Containers both allow you to download and deploy the code for development. You can also continue with local development.
61+
62+
### Quick Start in Codespaces
63+
1. Open in GitHub Codespaces (button above)
64+
2. Wait for the environment to load
65+
3. Run the following commands in the terminal:
66+
```bash
67+
azd up
68+
```
69+
4. Follow the prompts to select your Azure subscription and region
70+
5. Wait for deployment to complete (5-20 minutes) - you'll get a web app URL when finished
71+
72+
For detailed deployment options, see the [full deployment guide](./docs/deployment.md).
6173

6274
**After deployment, try these [sample questions](./docs/sample_questions.md) to test your agent.**
6375

76+
## Agent Evaluation
77+
78+
AI Foundry offers a number of [built-in evaluators](https://learn.microsoft.com/en-us/azure/ai-foundry/how-to/develop/agent-evaluate-sdk) to measure the quality, efficiency, risk and safety of your agents. For example, intent resolution, tool call accuracy, and task adherence evaluators are targeted to assess the performance of agent workflow, while content safety evaluator checks for inappropriate content in the responses such as violence or hate.
79+
80+
In this template, we show how these evaluations can be performed during different phases of your development cycle.
81+
82+
- **Local development**: You can use this [local evaluation script](evals/evaluate.py) to get performance and evaluation metrics based on a set of [test queries](evals/eval-queries.json) for a sample set of built-in evaluators.
83+
84+
The script reads the following environment variables:
85+
- `AZURE_EXISTING_AIPROJECT_ENDPOINT`: AI Project endpoint
86+
- `AZURE_EXISTING_AGENT_ID`: AI Agent Id, with fallback logic to look up agent Id by name `AZURE_AI_AGENT_NAME`
87+
- `AZURE_AI_AGENT_DEPLOYMENT_NAME`: Deployment model used by the AI-assisted evaluators, with fallback logic to your agent model
88+
89+
To install required packages and run the script:
90+
91+
```shell
92+
python -m pip install -r src/requirements.txt
93+
python -m pip install azure-ai-evaluation
94+
95+
python evals/evaluate.py
96+
```
97+
98+
- **Monitoring**: When tracing is enabled, the [application code](src/api/routes.py) sends an asynchronous evaluation request after processing a thread run, allowing continuous monitoring of your agent. You can view results from the AI Foundry Tracing tab.
99+
![Tracing](docs/images/tracing_eval_screenshot.png)
100+
Alternatively, you can go to your Application Insights logs for an interactive experience. Here is an example query to see logs on thread runs and related events.
101+
102+
```kql
103+
let thread_run_events = traces
104+
| extend thread_run_id = tostring(customDimensions.["gen_ai.thread.run.id"]);
105+
dependencies
106+
| extend thread_run_id = tostring(customDimensions.["gen_ai.thread.run.id"])
107+
| join kind=leftouter thread_run_events on thread_run_id
108+
| where isnotempty(thread_run_id)
109+
| project timestamp, thread_run_id, name, success, duration, event_message = message, event_dimensions=customDimensions1
110+
```
111+
112+
- **Continuous Integration**: You can try the [AI Agent Evaluation GitHub action](https://github.com/microsoft/ai-agent-evals) using the [sample GitHub workflow](.github/workflows/ai-evaluation.yaml) in your CI/CD pipeline. This GitHub action runs a set of queries against your agent, performs evaluations with evaluators of your choice, and produce a summary report. It also supports a comparison mode with statistical test, allowing you to iterate agent changes on your production environment with confidence. See [documentation](https://github.com/microsoft/ai-agent-evals) for more details.
113+
64114

65115
## Other Features
66116
Once you have the agents and the web app working, you are encouraged to try one of the following:

‎docs/other_features.md‎

Lines changed: 0 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -14,44 +14,6 @@ After accessing you resource group in Azure portal, choose your container app fr
1414

1515
You can view the App Insights tracing in Azure AI Foundry. Select your project on the Azure AI Foundry page and then click 'Tracing'.
1616

17-
## Agent Evaluation
18-
19-
AI Foundry offers a number of [built-in evaluators](https://learn.microsoft.com/en-us/azure/ai-foundry/how-to/develop/agent-evaluate-sdk) to measure the quality, efficiency, risk and safety of your agents. For example, intent resolution, tool call accuracy, and task adherence evaluators are targeted to assess the performance of agent workflow, while content safety evaluator checks for inappropriate content in the responses such as violence or hate.
20-
21-
In this template, we show how these evaluations can be performed during different phases of your development cycle.
22-
23-
- **Local development**: You can use this [local evaluation script](../evals/evaluate.py) to get performance and evaluation metrics based on a set of [test queries](../evals/eval-queries.json) for a sample set of built-in evaluators.
24-
25-
The script reads the following environment variables:
26-
- `AZURE_EXISTING_AIPROJECT_ENDPOINT`: AI Project endpoint
27-
- `AZURE_EXISTING_AGENT_ID`: AI Agent Id, with fallback logic to look up agent Id by name `AZURE_AI_AGENT_NAME`
28-
- `AZURE_AI_AGENT_DEPLOYMENT_NAME`: Deployment model used by the AI-assisted evaluators, with fallback logic to your agent model
29-
30-
To install required packages and run the script:
31-
32-
```shell
33-
python -m pip install -r src/requirements.txt
34-
python -m pip install azure-ai-evaluation
35-
36-
python evals/evaluate.py
37-
```
38-
39-
- **Monitoring**: When tracing is enabled, the [application code](../src/api/routes.py) sends an asynchronous evaluation request after processing a thread run, allowing continuous monitoring of your agent. You can view results from the AI Foundry Tracing tab.
40-
![Tracing](./images/tracing_eval_screenshot.png)
41-
Alternatively, you can go to your Application Insights logs for an interactive experience. Here is an example query to see logs on thread runs and related events.
42-
43-
```kql
44-
let thread_run_events = traces
45-
| extend thread_run_id = tostring(customDimensions.["gen_ai.thread.run.id"]);
46-
dependencies
47-
| extend thread_run_id = tostring(customDimensions.["gen_ai.thread.run.id"])
48-
| join kind=leftouter thread_run_events on thread_run_id
49-
| where isnotempty(thread_run_id)
50-
| project timestamp, thread_run_id, name, success, duration, event_message = message, event_dimensions=customDimensions1
51-
```
52-
53-
- **Continuous Integration**: You can try the [AI Agent Evaluation GitHub action](https://github.com/microsoft/ai-agent-evals) using the [sample GitHub workflow](../.github/workflows/ai-evaluation.yaml) in your CI/CD pipeline. This GitHub action runs a set of queries against your agent, performs evaluations with evaluators of your choice, and produce a summary report. It also supports a comparison mode with statistical test, allowing you to iterate agent changes on your production environment with confidence. See [documentation](https://github.com/microsoft/ai-agent-evals) for more details.
54-
5517
## AI Red Teaming Agent
5618

5719
The [AI Red Teaming Agent](https://learn.microsoft.com/azure/ai-foundry/concepts/ai-red-teaming-agent) is a powerful tool designed to help organizations proactively find security and safety risks associated with generative AI systems during design and development of generative AI models and applications.

0 commit comments

Comments
 (0)