Skip to content

Conversation

Wovchena
Copy link
Collaborator

@Wovchena Wovchena commented Oct 16, 2025

Description

WWB uses _OVMiniCPMVForCausalLM.preprocess_inputs() for optimum-intel and --hf. That function fails to apply chat_template for MiniCPM-o-2_6 and MiniCPM-V-2_6. A fix is required for optimum-intel. It will take time to validate both of the models with that fix. This PR addresses other identified issues. The original implementation applies chat_template: https://huggingface.co/openbmb/MiniCPM-o-2_6/blob/main/modeling_minicpmo.py#L959 but it's not used in WWB.
WWB results for --weights-format fp16 and unpublished fix for optimum-intel chat_template:

  1. --hf vs optimum-intel : 0.986911
  2. --hf vs --genai: 0.935212
  3. optimum-intel vs --genai 0.942425

Docs: https://wovchena.github.io/openvino.genai-public/docs/supported-models/#visual-language-models-vlms

That should be enough for this release but follow up PRs are needed. The ideal solution includes:

  1. Fix optimum-intel chat_template application
  2. Request WWB not to use optimum-intel with --hf
  3. Align --hf and optimum-intel (likely to be never completed because the results are close: 0.986911)
  4. Extend GenAI pre-commit tests
  5. Copy image resize implementation from PIL to GenAI
  6. Check if it's possible to align sin() and cos() with numpy with hardcoding commit sin() and cos() values

CVS-170987

Checklist:

  • Tests have been updated or added to cover the new code
  • This patch fully addresses the ticket.
  • I have made corresponding changes to the documentation

@Copilot Copilot AI review requested due to automatic review settings October 16, 2025 07:45
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This pull request fixes a bug in the WWB implementation where _OVMiniCPMVForCausalLM.preprocess_inputs() failed to apply chat templates for MiniCPM-o-2_6 and MiniCPM-V-2_6 models. The fix changes the image tag format from (<image>./</image>)\n to <image>./</image>\n across documentation and implementation files.

Key Changes

  • Updated image tag format for MiniCPM models from (<image>./</image>)\n to <image>./</image>\n
  • Added support for new MiniCPM-o-2_6 model variant
  • Fixed related algorithm implementations in the MiniCPM vision encoder

Reviewed Changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
tests/python_tests/test_vlm_pipeline.py Updated test to use correct image tag format without parentheses
src/python/py_vlm_pipeline.cpp Updated documentation strings to show correct image tag format for both MiniCPM variants
src/python/openvino_genai/py_openvino_genai.pyi Updated type stub documentation to reflect correct image tag format
src/cpp/src/visual_language/minicpm/classes.hpp Updated resample method signature to use single target size with padding parameter
src/cpp/src/visual_language/minicpm/classes.cpp Refactored implementation with improved algorithms and removed parentheses from image tag
src/cpp/src/visual_language/inputs_embedder.hpp Updated comment to reflect correct image tag format
src/cpp/src/debug_utils.hpp Added support for boolean numpy array type
src/cpp/include/openvino/genai/visual_language/pipeline.hpp Updated API documentation with correct image tag format
site/docs/use-cases/image-processing/_sections/_usage_options/index.mdx Updated documentation to separate MiniCPM-o-2_6 and MiniCPM-V-2_6 with correct tags
site/docs/supported-models/index.mdx Added notes about MiniCPM-o compatibility issues
site/docs/supported-models/_components/vlm-models-table/models.ts Added MiniCPM-o-2_6 model to supported models table
samples/cpp/visual_language_chat/README.md Removed specific model reference to make documentation more generic

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

@Wovchena Wovchena changed the title Fix mini cpm o 2 6 cvs 170987 Fix MiniCPM-o-2_6 Oct 16, 2025
@github-actions github-actions bot added category: visual language Visual language pipeline category: Python API Python API for GenAI category: CPP API Changes in GenAI C++ public headers no-match-files category: GH Pages Docs Github Pages documentation category: GGUF GGUF file reader category: VLM samples GenAI VLM samples labels Oct 16, 2025
@Wovchena Wovchena added this to the 2025.4 milestone Oct 16, 2025
@Wovchena Wovchena requested a review from yatarkan October 16, 2025 10:40
@Wovchena Wovchena enabled auto-merge October 16, 2025 11:51
@Wovchena Wovchena mentioned this pull request Oct 16, 2025
@Wovchena Wovchena added this pull request to the merge queue Oct 16, 2025
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Oct 16, 2025
@Wovchena Wovchena added this pull request to the merge queue Oct 17, 2025
Merged via the queue into openvinotoolkit:master with commit 939001f Oct 17, 2025
138 of 142 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

category: CPP API Changes in GenAI C++ public headers category: GGUF GGUF file reader category: GH Pages Docs Github Pages documentation category: Python API Python API for GenAI category: visual language Visual language pipeline category: VLM samples GenAI VLM samples Code Freeze no-match-files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants