Skip to content

Commit 59f55ea

Browse files
authored
docs: Add README for advanced image processing (#969)
1 parent 3a497d7 commit 59f55ea

File tree

4 files changed

+41
-0
lines changed

4 files changed

+41
-0
lines changed

docs/advanced_image_processing.md

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
# Advanced Image Processing
2+
3+
The Chat With Your Data solution accelerator supports advanced image processing using Azure AI Vision and GPT-4 Vision to obtain more meaningful data from images.
4+
5+
With advanced image processing enabled, GPT-4 vision is used to generate rich captions of images, and Azure Computer Vision is used to generate embeddings of images, in addition to the text embeddings generated from the image captions.
6+
7+
When querying data in Azure AI Search, vector search is performed using embeddings created for the search query using both Azure OpenAI and Azure Computer Vision.
8+
9+
Finally, if any image documents are retrieved from Azure AI Search, the images themselves are passed to the LLM as well as the text caption generated for the image.
10+
11+
All of this allows a user to ask questions about the meaning of images in the knowledge store, rather than just the text in an image.
12+
13+
Compare the results when uploading the following image:
14+
15+
![image](./images/architecture.png)
16+
17+
Without advanced image processing:
18+
19+
![image](./images/without_advanced_image_processing.png)
20+
21+
With advanced image processing:
22+
23+
![image](./images/with_advanced_image_processing.png)
24+
25+
## Enabling Advanced Image Processing
26+
27+
To enable advanced image processing:
28+
29+
- If deploying from the UI, set "Use Advanced Image Processing" to "true".
30+
31+
- If deploying using azd, run the following before deploying:
32+
33+
```bash
34+
azd env set USE_ADVANCED_IMAGE_PROCESSING true
35+
```
36+
37+
Once enabled, advanced image processing will be enabled for all supported image types, however it can be enabled/disabled for individual image types in the Admin app.
38+
39+
![image](./images/enable_advanced_image_processing.png)
40+
41+
Advanced image processing is only used in the `custom` conversation flow and not the `byod` flow, as Azure OpenAI On Your Data only supports Ada embeddings. It is currently not possible to use advanced image processing when integrated vectorization is enabled.
103 KB
Loading
180 KB
Loading
75.1 KB
Loading

0 commit comments

Comments
 (0)