Skip to content

Commit 2d41554

Browse files
committed
RDoc-3491 add the AI agents overview page a security concerns section
1 parent c89aba5 commit 2d41554

File tree

3 files changed

+133
-59
lines changed

3 files changed

+133
-59
lines changed

docs/ai-integration/ai-agents/ai-agents_overview.mdx

Lines changed: 105 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -16,12 +16,13 @@ import LanguageContent from "@site/src/components/LanguageContent";
1616
<Admonition type="note" title="">
1717

1818
* An **AI Agent** is an ongoing service that resides on a RavenDB server.
19-
After its creation by a RavenDB client, an agent can respond to client requests by mediating between the client, an AI model, and a RavenDB database.
19+
An agent can serve RavenDB clients by [mediating](../../ai-integration/ai-agents/ai-agents_overview#ai-agent-usage-flow-chart) between the clients, an AI model, and a RavenDB database.
2020

21-
* An AI agent can provide the AI model with a set of Query and Action tools.
22-
The AI model can then facilitate these tools and query the database or request the client to perform actions.
21+
* Through an AI agent, the LLM acquires the ability to query the database and request the client to perform actions.
2322

24-
* The client gains this way the ability to communicate with an AI model that has access to the database, and to easily automate complex workflows that leverage the AI model's insights and suggestions.
23+
* Granting the LLM access to a relevant data source like a company database can significantly enhance its ability to provide the client with accurate and context-aware responses, as well as reduce behaviors that harm its credibility, such as 'hallucinations' and user-pleasing bias.
24+
25+
* Clients can use an agent to automate complex workflows by leveraging AI capabilities like data analysis, decision-making, and understanding of natural language segments.
2526

2627
* In this article:
2728
* [Common use cases](../../ai-integration/ai-agents/ai-agents_overview#common-use-cases)
@@ -30,9 +31,8 @@ import LanguageContent from "@site/src/components/LanguageContent";
3031
* [Initiating a conversation](../../ai-integration/ai-agents/ai-agents_overview#initiating-a-conversation)
3132
* [AI agent usage flow chart](../../ai-integration/ai-agents/ai-agents_overview#ai-agent-usage-flow-chart)
3233
* [Streaming LLM responses (RavenDB 7.1.3 and up)](../../ai-integration/ai-agents/ai-agents_overview#streaming-llm-responses-ravendb-713-and-up)
33-
* [Security concerns](../../ai-integration/ai-agents/ai-agents_overview#security-concerns)
34-
* [AI agents and other AI features](../../ai-integration/ai-agents/ai-agents_overview#ai-agents-and-other-ai-features)
3534
* [Reducing throughput and expediting LLM response](../../ai-integration/ai-agents/ai-agents_overview#reducing-throughput-and-expediting-llm-response)
35+
* [Security concerns](../../ai-integration/ai-agents/ai-agents_overview#security-concerns)
3636

3737
</Admonition>
3838

@@ -44,7 +44,7 @@ AI agents are designed to easily integrate AI capabilities into applications and
4444

4545
* **Data analysis and reporting agents** can analyze large datasets to extract relevant data and present it in a user-friendly format, escalate customer issues and application output, create reports and highlight points of interest, and help businesses make informed decisions.
4646

47-
* **Automated content generation agents** can generate summaries, add automated comments to articles and and application-generated contents, reference readers to related material, and create marketing content based on user input and stored information.
47+
* **Automated content generation agents** can generate summaries, add automated comments to articles and application-generated content, reference readers to related material, and create marketing content based on user input and stored information.
4848

4949
* **Workflow automation agents** can automate repetitive tasks such as email sorting, spam filtering, form filling, or file organization.
5050

@@ -54,8 +54,14 @@ AI agents are designed to easily integrate AI capabilities into applications and
5454

5555
## Defining and running an AI agent
5656

57-
An AI agent is defined by a client and runs on a RavenDB server.
58-
Once defined, the agent can be invoked by the client to handle user requests, respond to events tracked by the client, and so on.
57+
An AI agent is an ongoing AI task that can be created by RavenDB clients (providing they have database administration permissions) and reside on a RavenDB server.
58+
Agents can be invoked by clients to, for example, handle user requests and respond to events tracked by the client.
59+
60+
```
61+
An agent can serve multiple clients concurrently.
62+
The agent configuration, logic, and tools will be shared by all its clients.
63+
Each client will maintain a different conversation instance and will be able to provide its own parameters, conduct its own conversation, and get its separate results when the conversation ends.
64+
```
5965

6066
<Admonition type="note" title="">
6167
* [Learn to create an AI agent using the client API](../../ai-integration/ai-agents/creating-ai-agents/creating-ai-agents_api)
@@ -66,16 +72,22 @@ Once defined, the agent can be invoked by the client to handle user requests, re
6672
To define an AI agent, the client needs to specify -
6773

6874
* A **connection string** to the AI model.
75+
[Create a connection string using the API](../../ai-integration/ai-agents/creating-ai-agents/creating-ai-agents_api#creating-a-connection-string)
76+
[Create a connection string using Studio](../../ai-integration/ai-agents/creating-ai-agents/creating-ai-agents_studio#configure-basic-settings)
77+
78+
* An **agent configuration** that defines the agent.
79+
[Define an agent configuration using the API](../../ai-integration/ai-agents/creating-ai-agents/creating-ai-agents_api#defining-an-agent-configuration)
80+
[Define an agent configuration using Studio](../../ai-integration/ai-agents/creating-ai-agents/creating-ai-agents_studio#configure-basic-settings)
6981

70-
* An **agent configuration** that includes -
82+
An agent configuration includes -
7183
* **Basic agent settings**, like the unique ID by which the system recognizes the task.
7284
* A **system prompt**, that defines AI model characteristics like its role.
7385
* Optional **agent parameters**.
7486
Agent parameters' values are provided by the client when it starts a conversation with the agent, and can be used in queries initiated by the LLM (see **query tools** below).
7587
* <a id="query-tools"/> Optional **query tools**.
7688
The LLM will be able to invoke query tools freely to retrieve data from the database.
77-
* **Read only operations**
78-
Query tools are only allowed to apply **read operations**.
89+
* **Read-only operations**
90+
Query tools can apply **read operations** only.
7991
To make changes in the database, use [action tools](../../ai-integration/ai-agents/ai-agents_overview#action-tools).
8092
* **Database access**
8193
The LLM has no direct access to the database. To use a query tool, it must send a query request to the agent, which will send the RQL query defined by the tool to the database and pass its results to the LLM.
@@ -96,6 +108,11 @@ To define an AI agent, the client needs to specify -
96108
The LLM will be able to use these tools to request the client to perform actions.
97109

98110
### Initiating a conversation:
111+
A conversation is a communication session between the client, the agent, and the LLM,
112+
during which the LLM may trigger agent tools to interact with the database and the client.
113+
[Initiate a conversation using the API](../../ai-integration/ai-agents/creating-ai-agents/creating-ai-agents_api#managing-conversations)
114+
[Initiate a conversation using Studio](../../ai-integration/ai-agents/creating-ai-agents/creating-ai-agents_studio#start-new-chat)
115+
99116
To start a conversation with the LLM, the agent will send it an **initial context** that includes -
100117

101118
* Pre-defined [agent configuration](../../ai-integration/ai-agents/ai-agents_overview#the-main-stages-in-defining-an-ai-agent) elements (automatically sent by the agent):
@@ -127,7 +144,7 @@ To start a conversation with the LLM, the agent will send it an **initial contex
127144

128145
* Optional **conversation history**
129146
To continue a conversation with the LLM, the agent will need to send it the entire history of the conversation so far.
130-
Conversations are automatically kept in documents in the `@conversations` collection. The client will need to reference the agent to the conversation that it wants to continue.
147+
Conversations are automatically kept in documents in the `@conversations` collection. The client will need to reference the agent to the conversation it wants to continue.
131148

132149
* A **user prompt**, set by the client, that defines this part of the conversation.
133150
The user prompt may be, for example, a question or a request for particular information.
@@ -190,39 +207,96 @@ Streaming can ease the processing of lengthy LLM responses for clients, and crea
190207

191208
Streaming is supported by most AI models, including OpenAI services like GPT-4 and Ollama models.
192209

193-
[Learn how to stream LLM responses using the API](../../ai-integration/ai-agents/creating-ai-agents/creating-ai-agents_api#stream-llm-responses)
210+
[Streaming LLM responses using the API](../../ai-integration/ai-agents/creating-ai-agents/creating-ai-agents_api#stream-llm-responses)
194211

195212
<hr />
196213

197-
## Security concerns
214+
## Reducing throughput and expediting LLM response
215+
216+
If throughput and LLM response time are a consideration, consider these options:
198217

199-
https://issues.hibernatingrhinos.com/issue/RavenDB-24777/AI-Agent-Security-Concerns
218+
### Set maximum number of querying iterations:
200219

201-
Though in our example the LLM helps us find and reward productive employees, we remain careful throughout the code not to provide it with personal employee details or proprietary company information.
220+
You can limit the number of times that the LLM is allowed to trigger database queries in response to a single user prompt.
202221

222+
[Setting iterations limit using the API](../../ai-integration/ai-agents/creating-ai-agents/creating-ai-agents_api#set-maximum-number-of-iterations)
203223

204-
<hr />
224+
### Define a chat trimming configuration:
205225

206-
## AI agents and other AI features
226+
The LLM doesn't keep conversations history. To allow a continuous conversation, the agent includes in every new message it sends to the LLM the history of the entire conversation since it started.
207227

208-
### AI agents and vector search
228+
To save traffic and tokens, you can summarize conversations history using **chat trimming**. This can be helpful when transfer rate and cost are a concern or the context becomes too large to handle efficiently.
229+
230+
[Configuring chat trimming using the API](../../ai-integration/ai-agents/creating-ai-agents/creating-ai-agents_api#set-chat-trimming-configuration)
231+
[Configuring chat trimming using Studio](../../ai-integration/ai-agents/creating-ai-agents/creating-ai-agents_studio#configure-chat-trimming)
232+
233+
### Optimize query tools:
234+
235+
When creating query tools -
236+
* Provide the LLM with clear instructions on how to use each query tool effectively.
237+
* Narrow your queries:
238+
* Design queries to return only the data that is relevant to the agent's role and the user's prompt.
239+
* You can limit the scope of a query both in the RQL statement itself and by using agent parameters to filter results.
240+
* Avoid overly broad queries that return large datasets, as they can overwhelm the LLM and lead to slower response times.
241+
* Consider setting a limit on the number of results returned by each query to prevent excessive data transfer and processing.
242+
* Supervise querying:
243+
* Test query tools with various prompts and scenarios to identify and address any performance bottlenecks.
244+
* Monitor the performance of query tools in production to identify and address any issues that arise over time.
245+
* Regularly review and update query tools to ensure they remain relevant and efficient as the database evolves.
246+
247+
[Creating query tools using the API](../../ai-integration/ai-agents/creating-ai-agents/creating-ai-agents_api#query-tools)
248+
[Creating query tools using Studio](../../ai-integration/ai-agents/creating-ai-agents/creating-ai-agents_studio#add-query-tools)
209249

210250
<hr />
211251

212-
## Reducing throughput and expediting LLM response
252+
## Security concerns
213253

214-
If throughput and LLM response time are a consideration, consider these options:
254+
### Concern: Unauthorized access to databases can lead to data breaches
215255

216-
### maximum number of querying iterations:
256+
* **Mitigation: Read-only access**
257+
The LLM has no direct access to the database. It can only request the agent, via query tools, to query the database on its behalf, and the agent can only apply read-only operations.
217258

218-
You can limit the number of times that the LLM is allowed to trigger database queries in response to a single user prompt.
219-
* [Set iterations limit using the API](../../ai-integration/ai-agents/creating-ai-agents/creating-ai-agents_api#set-maximum-number-of-iterations)
259+
* **Mitigation: DBA control**
260+
Control over the database is determined using certificates. Only users whose certificates grant them a database administrator or a higher role can create and manage agents.
261+
The DBA retains full control over connections to the AI model (through connection strings), the agent configuration, and the queries that the agent is allowed to run.
262+
263+
* **Mitigation: Agent scope**
264+
An AI agent is created for a specific database and has no access to other databases on the server, ensuring database-level isolation.
220265

221-
### Chat trimming configuration:
266+
### Concern: Data may be compromised during transition
222267

223-
The LLM doesn't keep the history of previous conversations. To allow a continuous conversation, we include in every new message we send to the LLM the history of the entire conversation since its start.
224-
To save traffic and tokens, you can summarize conversations history. This can be helpful when transfer rate and cost are a concern or the context may become too large to handle efficiently.
268+
* **Mitigation: Secure TLS (Transport Layer Security) communication**
269+
All data is transferred over HTTPS between the client, the agent, the database, and the AI model, to ensure its encryption during transit.
225270

226-
* [Configure chat trimming using the API](../../ai-integration/ai-agents/creating-ai-agents/creating-ai-agents_api#set-chat-trimming-configuration)
227-
* [Configure chat trimming using Studio](../../ai-integration/ai-agents/creating-ai-agents/creating-ai-agents_studio#configure-chat-trimming)
228-
271+
### Concern: Inability to trace malicious or unexpected actions related to agents
272+
273+
* **Mitigation: Audit logging**
274+
RavenDB [admin logs](../../studio/server/debug/admin-logs/) track the creation, modification, and deletion of AI agents, as well as agent interactions with the database.
275+
276+
Example of an audit log entry recorded when an agent was deleted:
277+
```
278+
Starting to process record 16 (current 15) for aiAgent_useHandleToRunChat_1.
279+
Type: DeleteAiAgentCommand.
280+
Cluster database change type: RecordChanged
281+
Date 2025-09-23 22:29:45.0391
282+
Level DEBUG
283+
Thread ID 58
284+
Resource aiAgent_useHandleToRunChat_1
285+
Logger Raven.Server.Documents.DocumentDatabase
286+
```
287+
### Concern: Sensitive data might inadvertently be memorized and reproduced by the AI model
288+
289+
* **Mitigation: Free selection of AI model**
290+
RavenDB doesn't enforce the usage of specific providers or AI models, but gives you free choice of the services that best suit your needs and security requirements.
291+
When using the service of your choice, it is your responsibility to define safe queries and expose only the data that it is in your interest to share with the AI model.
292+
293+
* **Mitigation: Agent parameters**
294+
You can use [agent parameters](../../ai-integration/ai-agents/ai-agents_overview#query-parameters) to limit the scope of the defined query and the dataset subsequently transferred to the AI model.
295+
296+
### Concern: Validation or injection attacks crafted through malicious user input
297+
298+
* **Mitigation: Query scope**
299+
The agent queries a limited subset of the stored data, restricting an attacker's access to the rest of the data and to data belonging to other users.
300+
301+
* **Mitigation: Read-only access**
302+
Query tools can apply read-only RQL queries, preventing attackers from modifying any data.

0 commit comments

Comments
 (0)