Skip to content

Commit 0f738cd

Browse files
howieleunghowieleung
authored
Merge pull request #172 from Azure-Samples/howie/doc-update
Update doc
2 parents d403ad4 + 40d880a commit 0f738cd

File tree

2 files changed

+49
-41
lines changed

2 files changed

+49
-41
lines changed

‎README.md‎

Lines changed: 49 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ The agent leverages the Azure AI Agent service and utilizes file search for know
44

55
<div style="text-align:center;">
66

7-
[**SOLUTION OVERVIEW**](#solution-overview) \| [**GETTING STARTED**](#getting-started) \| [**OTHER FEATURES**](#other-features) \| [**RESOURCE CLEAN-UP**](#resource-clean-up) \| [**GUIDANCE**](#guidance) \| [**TROUBLESHOOTING**](#troubleshooting)
7+
[**SOLUTION OVERVIEW**](#solution-overview) \| [**GETTING STARTED**](#getting-started) \| [**AGENT EVALUATION**](#agent-evaluation) \| [**OTHER FEATURES**](#other-features) \| [**RESOURCE CLEAN-UP**](#resource-clean-up) \| [**GUIDANCE**](#guidance) \| [**TROUBLESHOOTING**](#troubleshooting)
88

99
</div>
1010

@@ -52,15 +52,61 @@ Here is a screenshot showing the chatting web application with requests and resp
5252

5353
## Getting Started
5454

55-
### Quick Deploy
55+
### Quick Start
5656

5757
| [![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/Azure-Samples/get-started-with-ai-agents) | [![Open in Dev Containers](https://img.shields.io/static/v1?style=for-the-badge&label=Dev%20Containers&message=Open&color=blue&logo=visualstudiocode)](https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/Azure-Samples/get-started-with-ai-agents) |
5858
|---|---|
5959

60-
Github Codespaces and Dev Containers both allow you to download and deploy the code for development. You can also continue with local development. Once you have selected your environment, [click here to launch the development and deployment guide](./docs/deployment.md)
60+
1. Click `Open in GitHub Codespaces` or `Dev Containers` button above
61+
2. Wait for the environment to load
62+
3. Run the following commands in the terminal:
63+
```bash
64+
azd up
65+
```
66+
4. Follow the prompts to select your Azure subscription and region
67+
5. Wait for deployment to complete (5-20 minutes) - you'll get a web app URL when finished
6168

69+
For detailed deployment options and troubleshooting, see the [full deployment guide](./docs/deployment.md).
6270
**After deployment, try these [sample questions](./docs/sample_questions.md) to test your agent.**
6371

72+
## Agent Evaluation
73+
74+
AI Foundry offers a number of [built-in evaluators](https://learn.microsoft.com/en-us/azure/ai-foundry/how-to/develop/agent-evaluate-sdk) to measure the quality, efficiency, risk and safety of your agents. For example, intent resolution, tool call accuracy, and task adherence evaluators are targeted to assess the performance of agent workflow, while content safety evaluator checks for inappropriate content in the responses such as violence or hate.
75+
76+
In this template, we show how these evaluations can be performed during different phases of your development cycle.
77+
78+
- **Local development**: You can use this [local evaluation script](evals/evaluate.py) to get performance and evaluation metrics based on a set of [test queries](evals/eval-queries.json) for a sample set of built-in evaluators.
79+
80+
The script reads the following environment variables:
81+
- `AZURE_EXISTING_AIPROJECT_ENDPOINT`: AI Project endpoint
82+
- `AZURE_EXISTING_AGENT_ID`: AI Agent Id, with fallback logic to look up agent Id by name `AZURE_AI_AGENT_NAME`
83+
- `AZURE_AI_AGENT_DEPLOYMENT_NAME`: Deployment model used by the AI-assisted evaluators, with fallback logic to your agent model
84+
85+
To install required packages and run the script:
86+
87+
```shell
88+
python -m pip install -r src/requirements.txt
89+
python -m pip install azure-ai-evaluation
90+
91+
python evals/evaluate.py
92+
```
93+
94+
- **Monitoring**: When tracing is enabled, the [application code](src/api/routes.py) sends an asynchronous evaluation request after processing a thread run, allowing continuous monitoring of your agent. You can view results from the AI Foundry Tracing tab.
95+
![Tracing](docs/images/tracing_eval_screenshot.png)
96+
Alternatively, you can go to your Application Insights logs for an interactive experience. Here is an example query to see logs on thread runs and related events.
97+
98+
```kql
99+
let thread_run_events = traces
100+
| extend thread_run_id = tostring(customDimensions.["gen_ai.thread.run.id"]);
101+
dependencies
102+
| extend thread_run_id = tostring(customDimensions.["gen_ai.thread.run.id"])
103+
| join kind=leftouter thread_run_events on thread_run_id
104+
| where isnotempty(thread_run_id)
105+
| project timestamp, thread_run_id, name, success, duration, event_message = message, event_dimensions=customDimensions1
106+
```
107+
108+
- **Continuous Integration**: You can try the [AI Agent Evaluation GitHub action](https://github.com/microsoft/ai-agent-evals) using the [sample GitHub workflow](.github/workflows/ai-evaluation.yaml) in your CI/CD pipeline. This GitHub action runs a set of queries against your agent, performs evaluations with evaluators of your choice, and produce a summary report. It also supports a comparison mode with statistical test, allowing you to iterate agent changes on your production environment with confidence. See [documentation](https://github.com/microsoft/ai-agent-evals) for more details.
109+
64110

65111
## Other Features
66112
Once you have the agents and the web app working, you are encouraged to try one of the following:

‎docs/other_features.md‎

Lines changed: 0 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -14,44 +14,6 @@ After accessing you resource group in Azure portal, choose your container app fr
1414

1515
You can view the App Insights tracing in Azure AI Foundry. Select your project on the Azure AI Foundry page and then click 'Tracing'.
1616

17-
## Agent Evaluation
18-
19-
AI Foundry offers a number of [built-in evaluators](https://learn.microsoft.com/en-us/azure/ai-foundry/how-to/develop/agent-evaluate-sdk) to measure the quality, efficiency, risk and safety of your agents. For example, intent resolution, tool call accuracy, and task adherence evaluators are targeted to assess the performance of agent workflow, while content safety evaluator checks for inappropriate content in the responses such as violence or hate.
20-
21-
In this template, we show how these evaluations can be performed during different phases of your development cycle.
22-
23-
- **Local development**: You can use this [local evaluation script](../evals/evaluate.py) to get performance and evaluation metrics based on a set of [test queries](../evals/eval-queries.json) for a sample set of built-in evaluators.
24-
25-
The script reads the following environment variables:
26-
- `AZURE_EXISTING_AIPROJECT_ENDPOINT`: AI Project endpoint
27-
- `AZURE_EXISTING_AGENT_ID`: AI Agent Id, with fallback logic to look up agent Id by name `AZURE_AI_AGENT_NAME`
28-
- `AZURE_AI_AGENT_DEPLOYMENT_NAME`: Deployment model used by the AI-assisted evaluators, with fallback logic to your agent model
29-
30-
To install required packages and run the script:
31-
32-
```shell
33-
python -m pip install -r src/requirements.txt
34-
python -m pip install azure-ai-evaluation
35-
36-
python evals/evaluate.py
37-
```
38-
39-
- **Monitoring**: When tracing is enabled, the [application code](../src/api/routes.py) sends an asynchronous evaluation request after processing a thread run, allowing continuous monitoring of your agent. You can view results from the AI Foundry Tracing tab.
40-
![Tracing](./images/tracing_eval_screenshot.png)
41-
Alternatively, you can go to your Application Insights logs for an interactive experience. Here is an example query to see logs on thread runs and related events.
42-
43-
```kql
44-
let thread_run_events = traces
45-
| extend thread_run_id = tostring(customDimensions.["gen_ai.thread.run.id"]);
46-
dependencies
47-
| extend thread_run_id = tostring(customDimensions.["gen_ai.thread.run.id"])
48-
| join kind=leftouter thread_run_events on thread_run_id
49-
| where isnotempty(thread_run_id)
50-
| project timestamp, thread_run_id, name, success, duration, event_message = message, event_dimensions=customDimensions1
51-
```
52-
53-
- **Continuous Integration**: You can try the [AI Agent Evaluation GitHub action](https://github.com/microsoft/ai-agent-evals) using the [sample GitHub workflow](../.github/workflows/ai-evaluation.yaml) in your CI/CD pipeline. This GitHub action runs a set of queries against your agent, performs evaluations with evaluators of your choice, and produce a summary report. It also supports a comparison mode with statistical test, allowing you to iterate agent changes on your production environment with confidence. See [documentation](https://github.com/microsoft/ai-agent-evals) for more details.
54-
5517
## AI Red Teaming Agent
5618

5719
The [AI Red Teaming Agent](https://learn.microsoft.com/azure/ai-foundry/concepts/ai-red-teaming-agent) is a powerful tool designed to help organizations proactively find security and safety risks associated with generative AI systems during design and development of generative AI models and applications.

0 commit comments

Comments
 (0)