[ChatQnA Core] Ollama Integration #890

hteeyeoh · 2025-09-03T07:47:22Z

Description

Integrate Ollama framework into ChatQnA Core. User can choose to download model from from Ollama and serve them using Ollama.

Fixes # (issue)

Any Newly Introduced Dependencies

ollama binary
langchain-ollama

How Has This Been Tested?

local dev system and development kubernetes cluster. Unit test have been included using pytest framework

Checklist:

I agree to use the APACHE-2.0 license for my code changes.
I have not introduced any 3rd party components incompatible with APACHE-2.0.
I have not included any company confidential information, trade secret, password or security token.
I have performed a self-review of my code.

Signed-off-by: Yeoh, Hoong Tee <[email protected]>

Update packages to fix vulnerability Signed-off-by: Yeoh, Hoong Tee <[email protected]>

Signed-off-by: Yeoh, Hoong Tee <[email protected]>

Change opt

Signed-off-by: Yeoh, Hoong Tee <[email protected]>

Image Updates nginx & ovms

bharagha

Have not checked the config files and test files in detail yet

bharagha · 2025-09-09T03:56:09Z

sample-applications/chat-question-and-answer-core/app/chain.py

-    download_huggingface_model(config.EMBEDDING_MODEL_ID, config._CACHE_DIR)
-    download_huggingface_model(config.RERANKER_MODEL_ID, config._CACHE_DIR)
-    download_huggingface_model(config.LLM_MODEL_ID, config._CACHE_DIR)
+    elif config.MODEL_BACKEND == "ollama":


Thinking out aloud. Can we use runtime instead of backend? Or a better name
Reason being Ollama can have OpenVINO backend too though that is not the case currently.

Sure we can rename that.

Changes updated.

bharagha · 2025-09-09T03:58:47Z

sample-applications/chat-question-and-answer-core/app/chain.py

-            search_kwargs={"k": 3, "score_threshold": 0.5}, search_type=search_method
+            search_kwargs={
+                "k": 3,
+                "fetch_k": fetch_k,


Do we have any empirical number on the quality of retrieval with threshold vs. top-k strategy?

both are finds and now i try to align with what modular does to keep the design same.

bharagha · 2025-09-09T07:33:40Z

sample-applications/chat-question-and-answer-core/app/chain.py

+    else:
+        raise ValueError(f"Unsupported model backend: {config.MODEL_BACKEND}")

+    embedding, llm, reranker = backend_instance.init_models()


What about initialization of llm, embedding, reranker in test mode?

In test mode, initial design was model download, conversion, and initialization will be mocked via pytest. So, in test mode, we are not gonna actually to download the model, doing the conversion and initialization. Those processes will be mocked.

bharagha · 2025-09-09T07:36:50Z

sample-applications/chat-question-and-answer-core/app/config.py

-            if not model_id:
-                raise ValueError(f"{model_name} must not be an empty string.")
+
+    def _validate_backend_settings(self):


The validate function is common across. Maintenance etc. becomes tougher on a continuous basis. Is it instead possible to have validate as part of backend specific configuration class?

Ya we can will try to figure this out.

changes updated. implemented a validator class to include those validators in corresponding runtime. In future, the new runtime can have their validator implemented under the class.

bharagha · 2025-09-09T07:41:35Z

sample-applications/chat-question-and-answer-core/app/ollama_backend.py

+        # Set the `OLLAMA_MODELS` to store the Ollama models
+        os.environ['OLLAMA_MODELS'] = f"{self.cache_dir}/ollama_models"
+
+        try:


Is the check for already active server part of Ollama handling? If not, should we not add some logic for the same? Also ensure no zombie process is left behind w.r.t cleanup etc.

ollama server will only start once at the beginning of the application. The process is spawn within the container so when container is terminated the process will terminated as well.

bharagha · 2025-09-09T08:46:08Z

sample-applications/chat-question-and-answer-core/app/ollama_backend.py

+from datetime import datetime, timezone
+import ollama
+import os
+import re


Is it used?

ya re is used in line 91 in exception block to clean the ANSI codes from the ollama server error. This is to give a better view for the error if anything happens/error when starting the ollama server

bharagha · 2025-09-09T08:48:01Z

sample-applications/chat-question-and-answer-core/app/ollama_routes.py

+
+@router.get("/ollama-model", tags=["Model API"], summary="Get OLLAMA model metadata")
+async def get_ollama_model_metadata(model_id: str = ""):
+    """


What is model_id is ""? Are we setting to a non-null value in config file?

No. this is the default value for the endpoint. And this is just some utilities route for specific to ollama to get or show model metadata detail. Instead of go into the container and run 'ollama ps' or 'ollama show <model_id>' cli command. I provide these routes just for debug and utitlity purpose.

bharagha · 2025-09-09T08:51:24Z

sample-applications/chat-question-and-answer-core/app/openvino_backend.py

+        model_path = os.path.join(cache_dir, model_id)
+
+        if os.path.isdir(model_path):
+            logger.info(f"Optimized {model_id} exists in {cache_dir}. Skipping conversion...")


What if it was not converted correctly? Is just a folder presence sufficient?
Check if you can place a marker at end of successful conversion or checksum based check.

bharagha · 2025-09-09T08:55:29Z

sample-applications/chat-question-and-answer-core/chart/subchart/chatqna-ui/values.yaml

-  uiTag: "core_1.2.2"
  pullPolicy: IfNotPresent
+  tags:
+    ui: "core_1.3.0"


What change in the UI?

nginx image upgrade and path change from /usr/ to /opt/ for nginx config storing. Also UI package vulnerabilities fix.

Signed-off-by: Yeoh, Hoong Tee <[email protected]>

madhuri-rai07 · 2025-09-26T06:18:57Z

ChatQna modular changes are already merged, please remove them @hteeyeoh

hteeyeoh · 2025-09-26T06:25:01Z

ChatQna modular changes are already merged, please remove them @hteeyeoh

Done. Resolved the conflict.

14pankaj and others added 16 commits August 4, 2025 16:00

Fix: Permission issue for EMT helm

03bbea3

update: changed to opt

1628cb2

[ChatQnA-Core] Ollama integration

033e15f

Signed-off-by: Yeoh, Hoong Tee <[email protected]>

[ChatQnA-Core] Helm Chart for Ollama integration

3e69d76

Signed-off-by: Yeoh, Hoong Tee <[email protected]>

Merge branch 'open-edge-platform:main' into ollama

60dfe07

[ChatQnA-Core] Unit test for both openvino and ollama

5be48af

Signed-off-by: Yeoh, Hoong Tee <[email protected]>

[ChatQnA-Core]: Update documentation

73b429e

Signed-off-by: Yeoh, Hoong Tee <[email protected]>

[ChatQnA-Core] Update ollama model endpoint

30ab643

Signed-off-by: Yeoh, Hoong Tee <[email protected]>

[ChatQnA-Core] Update ollama to latest version

1cdbbfe

Signed-off-by: Yeoh, Hoong Tee <[email protected]>

[ChatQnA Core] Include trust-remote-code flag during conversion

7c5b689

Signed-off-by: Yeoh, Hoong Tee <[email protected]>

fix: ITEP-71084 updated nginx image for sample apps

abd3e02

Update: nginx alipne image for stability

af7b358

[chatqna-core] Update dependencies packages

74ac839

Update packages to fix vulnerability Signed-off-by: Yeoh, Hoong Tee <[email protected]>

[chatqna-core-ui] Update dependencies and fix unittest

9787324

Signed-off-by: Yeoh, Hoong Tee <[email protected]>

[chatqna-core-ui] Update UI image tag

fd4da5b

Signed-off-by: Yeoh, Hoong Tee <[email protected]>

Merge branch 'open-edge-platform:main' into ollama

7b83a7c

hteeyeoh requested review from bharagha, yogeshmpandey, madhuri-rai07 and a team as code owners September 3, 2025 07:47

14pankaj and others added 6 commits September 4, 2025 09:13

host path to OPT folder

bd36f39

Change opt

Feat: Upgraded OVMS and deployment step changes

acf48ed

fix:updated folder names

88abdf8

fea: Debian nginx images for sample apps

de68af6

Merge branch 'open-edge-platform:main' into ollama

e497e02

[chatqna-ui] Bump NGINX version to 1.29.1

d8abf0c

Signed-off-by: Yeoh, Hoong Tee <[email protected]>

hteeyeoh force-pushed the ollama branch from 5086e15 to d8abf0c Compare September 8, 2025 04:18

14pankaj added 3 commits September 9, 2025 09:21

feat: nginx Image update

63646b9

fix: minor updates

7a2edd1

fix: Updated helm with ovms image & script updates.

3780832

14pankaj and others added 4 commits September 9, 2025 10:19

minor updates

d79d059

Merge branch 'ollama' into PenTestFxies

ddc0f0f

Merge pull request #3 from 14pankaj/PenTestFxies

59aed07

Image Updates nginx & ovms

Merge branch 'open-edge-platform:main' into ollama

13c6016

bharagha reviewed Sep 9, 2025

View reviewed changes

[chatqna-core] Rename config variable name and create validator class

23992cc

Signed-off-by: Yeoh, Hoong Tee <[email protected]>

hteeyeoh force-pushed the ollama branch from 1e92ba9 to 23992cc Compare September 10, 2025 02:01

14pankaj and others added 4 commits September 11, 2025 13:13

updated ovms to latest 2025.3

aa83c9b

Merge remote-tracking branch 'origin/main' into ollama

2ff26e5

Merge branch 'main' into ollama

4c5ffca

[chatqna-core]: Update helm deployment md

9011358

Signed-off-by: Yeoh, Hoong Tee <[email protected]>

madhuri-rai07 previously approved these changes Sep 26, 2025

View reviewed changes

Merge branch 'main' into ollama

3efa9b0

hteeyeoh dismissed madhuri-rai07’s stale review via 3efa9b0 September 26, 2025 06:23

hteeyeoh added 2 commits September 30, 2025 15:19

Merge branch 'open-edge-platform:main' into ollama

a829c59

Merge branch 'main' into ollama

fb7ed12

[ChatQnA Core] Ollama Integration #890

Are you sure you want to change the base?

[ChatQnA Core] Ollama Integration #890

Uh oh!

Conversation

hteeyeoh commented Sep 3, 2025

Description

Any Newly Introduced Dependencies

How Has This Been Tested?

Checklist:

Uh oh!

bharagha left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

madhuri-rai07 commented Sep 26, 2025

Uh oh!

hteeyeoh commented Sep 26, 2025

Uh oh!

Uh oh!