Using openvino-quantized embedder and reranker from huggingface hub

[chtanch]

For my application (which relies on openvino model server), I would like to use openvino-quantized models from huggingface hub and avoid doing the quantization step myself.

For chat llm models, eg, OpenVINO/Qwen2.5-7B-Instruct-int4-ov, I'll need an additional graph.pbtxt for ovms to work. It seems that I can use the same graph.pbtxt for all models, so I can include a pre-generated graph.pbtxt .

However, for embedder (and reranker) models, eg, OpenVINO/bge-base-en-v1.5-int8-ov, I'll need to include graph.pbtxt, openvino_detokenizer.bin, and openvino_detokenizer.xml. The tokenizer files seem to be model-dependent, so using pregenerated files is not reliable.

Is there a solution for using openvino-quantized embedder/reranker models from huggingface hub? Or do I have to quantize base models (eg, BAAI/bge-base-en-v1.5) myself with export_model.py

[dtrawins]

@chtanch Model https://huggingface.co/OpenVINO/bge-base-en-v1.5-int8-ov/tree/main have now added the missing tokenizer so you could be able to use it directly in OVMS https://docs.openvino.ai/2025/model-server/ovms_docs_serving_model.html#starting-the-genai-model-from-hugging-face-openvino-organization-directly

The following command can be used:

docker run -p 8000:8000 --user $(id -u):$(id -g) --rm -v $(pwd)/models:/models:rw openvino/model_server:2025.2.1  --model_repository_path /models --source_model OpenVINO/bge-base-en-v1.5-int8-ov --task embeddings --rest_port 8000

[chtanch]

@chtanch Model https://huggingface.co/OpenVINO/bge-base-en-v1.5-int8-ov/tree/main have now added the missing tokenizer so you could be able to use it directly in OVMS https://docs.openvino.ai/2025/model-server/ovms_docs_serving_model.html#starting-the-genai-model-from-hugging-face-openvino-organization-directly

The following command can be used:

docker run -p 8000:8000 --user $(id -u):$(id -g) --rm -v $(pwd)/models:/models:rw openvino/model_server:2025.2.1  --model_repository_path /models --source_model OpenVINO/bge-base-en-v1.5-int8-ov --task embeddings --rest_port 8000

Great, thank you!

[chtanch]

Can you help update the embedder and reranker models on modelscope as well? We are using those models for customers based in China

https://www.modelscope.cn/models/OpenVINO/bge-base-en-v1.5-int8-ov/files
https://www.modelscope.cn/models/OpenVINO/bge-reranker-base-int8-ov/files