[CI] [GHA] Use `OV_CACHE` in the WWB tests #2781

akashchi · 2025-10-01T11:07:11Z

No description provided.

as-suvorov · 2025-10-01T13:42:36Z

@akashchi any ideas about: https://github.com/openvinotoolkit/openvino.genai/actions/runs/18162007204/job/51697457952#step:7:4574

ERROR tools/who_what_benchmark/tests/test_cli_text.py::test_text_target_model - OSError: Can't load tokenizer for 'facebook/opt-125m'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'facebook/opt-125m' is the correct path to a directory containing all relevant files for a GPT2TokenizerFast tokenizer. ?

sgonorov · 2025-10-01T14:39:00Z

tools/who_what_benchmark/tests/test_cli_image.py

    ],
 )
 def test_image_model_types(model_id, model_type, backend, tmp_path):
+    MODEL_PATH = os.path.join(MODEL_CACHE, model_id.replace("/", "--"))


I think we should use Path syntax there, not os

sgonorov · 2025-10-01T14:39:53Z

tools/who_what_benchmark/tests/test_cli_image.py

 )
 def test_image_custom_dataset(model_id, model_type, backend, tmp_path):
    GT_FILE = tmp_path / "test_sd.csv"
+    MODEL_PATH = os.path.join(MODEL_CACHE, model_id.replace("/", "--"))


Same here - looks like it's better to use Path syntax here

sgonorov

Looks like a new Path intervenes with old string paths. Better check it once more.

akashchi · 2025-10-01T16:12:00Z

I believe that this

Looks like a new Path intervenes with old string paths. Better check it once more.

and the error reported by @as-suvorov are not because of the Path/os.path but because models were not present on the share and thus could not be used by path and firstly needed to be downloaded by model_id and then used by the full paths. I am working on it.

Copilot

Pull Request Overview

This PR implements caching functionality in the WHO WHAT BENCHMARK (WWB) tests by leveraging the OV_CACHE environment variable. The changes optimize test performance by reusing cached models across test runs instead of creating temporary directories each time.

Introduces centralized cache path configuration using OV_CACHE environment variable
Updates all test files to use shared cache directory instead of temporary paths
Adds conditional cleanup logic based on CLEANUP_CACHE environment variable

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
tools/who_what_benchmark/tests/constants.py	New file defining cache path and cleanup configuration
tools/who_what_benchmark/tests/test_cli_vlm.py	Updates VLM tests to use cached models and remove tmp_path dependency
tools/who_what_benchmark/tests/test_cli_text.py	Converts text tests from temporary directories to cache-based approach
tools/who_what_benchmark/tests/test_cli_image.py	Migrates image tests to use centralized cache directory

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

tools/who_what_benchmark/tests/constants.py

tests/python_tests/test_kv_cache_eviction.py

This reverts commit 72ab81a.

Copilot

Pull Request Overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-10-06T12:12:59Z

tools/who_what_benchmark/tests/test_cli_image.py

    GT_FILE = tmp_path / "gt.csv"
-    MODEL_PATH = os.path.join(MODEL_CACHE, model_id.replace("/", "--"))
+    MODEL_PATH = MODEL_CACHE.joinpath(model_id.replace("/", "--"))
+    MODEL_PATH = MODEL_PATH if MODEL_PATH.exists() else model_id


The fallback logic reassigns MODEL_PATH from a Path object to a string, creating inconsistent types. Consider using str(MODEL_PATH) in the conditional or ensuring consistent Path usage throughout.

Suggested change

MODEL_PATH = MODEL_PATH if MODEL_PATH.exists() else model_id

MODEL_PATH = str(MODEL_PATH) if MODEL_PATH.exists() else model_id

The underlying from_pretrained method accepts both str/Path types, so this should not be an issue.

Copilot · 2025-10-06T12:13:00Z

tools/who_what_benchmark/tests/test_cli_text.py

    run_wwb([
        "--base-model",
-        "Qwen/Qwen2-0.5B",
+        'Qwen/Qwen2-0.5B',


[nitpick] Inconsistent quote style. The surrounding code uses double quotes, but this line uses single quotes. Consider using double quotes for consistency.

Suggested change

'Qwen/Qwen2-0.5B',

"Qwen/Qwen2-0.5B",

tools/who_what_benchmark/tests/constants.py

tools/who_what_benchmark/tests/test_cli_vlm.py

tools/who_what_benchmark/tests/constants.py

tools/who_what_benchmark/tests/test_cli_image.py

tools/who_what_benchmark/tests/test_cli_text.py

tools/who_what_benchmark/tests/test_cli_vlm.py

Co-authored-by: Alexander Suvorov <[email protected]>

tools/who_what_benchmark/tests/test_cli_vlm.py

apaniukov · 2025-10-07T15:26:30Z

tools/who_what_benchmark/tests/test_cli_image.py

 def setup_module():
    for model_id in OV_IMAGE_MODELS:
-        MODEL_PATH = os.path.join(MODEL_CACHE, model_id.replace("/", "--"))
+        MODEL_PATH = MODEL_CACHE / model_id.replace("/", "--")


It seems that WWB and GenAI tests use different replacement strategies. .replace("/", "_") in GenAI

apaniukov · 2025-10-07T15:29:14Z

tools/who_what_benchmark/tests/constants.py

+        ov_cache = os.path.join(os.environ["OV_CACHE"], date_subfolder)
+        try:
+            optimum_intel_version = metadata.version("optimum-intel")
+            transformers_version = metadata.version("transformers")
+            ov_cache = os.path.join(ov_cache, f"optimum-intel-{optimum_intel_version}_transformers-{transformers_version}")


Suggested change

ov_cache = os.path.join(os.environ["OV_CACHE"], date_subfolder)

try:

optimum_intel_version = metadata.version("optimum-intel")

transformers_version = metadata.version("transformers")

ov_cache = os.path.join(ov_cache, f"optimum-intel-{optimum_intel_version}_transformers-{transformers_version}")

ov_cache = Path(os.environ["OV_CACHE"]) / date_subfolder

try:

optimum_intel_version = metadata.version("optimum-intel")

transformers_version = metadata.version("transformers")

ov_cache = ov_cache / f"optimum-intel-{optimum_intel_version}_transformers-{transformers_version}"

apaniukov · 2025-10-07T15:30:02Z

tools/who_what_benchmark/tests/constants.py

+        ov_cache = temp_dir.name
+    return Path(ov_cache) / "wwb_cache"


Suggested change

ov_cache = temp_dir.name

return Path(ov_cache) / "wwb_cache"

ov_cache = Path(temp_dir.name)

return ov_cache / "wwb_cache"

apaniukov · 2025-10-07T15:32:07Z

tools/who_what_benchmark/tests/constants.py

Could be a conftest.py. should_cleanup and wwb_cache_path could be session fixtures.

apaniukov · 2025-10-07T15:39:30Z

tools/who_what_benchmark/tests/test_cli_image.py

 def setup_module():
    for model_id in OV_IMAGE_MODELS:
-        MODEL_PATH = os.path.join(MODEL_CACHE, model_id.replace("/", "--"))
+        MODEL_PATH = MODEL_CACHE / model_id.replace("/", "--")
        subprocess.run(["optimum-cli", "export", "openvino", "--model", model_id, MODEL_PATH], capture_output=True, text=True)


 def teardown_module():
-    logger.info("Remove models")
-    shutil.rmtree(MODEL_CACHE)
+    if SHOULD_CLEANUP:
+        logger.info("Removing models")
+        shutil.rmtree(MODEL_CACHE)


Suggested change

def setup_module():

for model_id in OV_IMAGE_MODELS:

MODEL_PATH = os.path.join(MODEL_CACHE, model_id.replace("/", "--"))

MODEL_PATH = MODEL_CACHE / model_id.replace("/", "--")

subprocess.run(["optimum-cli", "export", "openvino", "--model", model_id, MODEL_PATH], capture_output=True, text=True)

def teardown_module():

logger.info("Remove models")

shutil.rmtree(MODEL_CACHE)

if SHOULD_CLEANUP:

logger.info("Removing models")

shutil.rmtree(MODEL_CACHE)

@pytest.fixture(scope="module", autouse=True)

def setup_module(wwb_cache_path, should_cleanup):

for model_id in OV_IMAGE_MODELS:

MODEL_PATH = wwb_cache_path / model_id.replace("/", "--")

subprocess.run(["optimum-cli", "export", "openvino", "--model", model_id, MODEL_PATH], capture_output=True, text=True)

yield

if should_cleanup:

logger.info("Removing models")

shutil.rmtree(wwb_cache_path)

as-suvorov · 2025-10-07T15:44:23Z

@apaniukov good comments. Don't you mind to address them in a separate PR? I want to merge this PR asap to unblock other PR

…nvino.genai into ci/gha/ov-cache-wwb

add WWB cache dir

6495cfd

akashchi added this to the 2025.4 milestone Oct 1, 2025

akashchi added the WIP label Oct 1, 2025

github-actions bot added the category: WWB PR changes WWB label Oct 1, 2025

sgonorov approved these changes Oct 1, 2025

View reviewed changes

akashchi added 2 commits October 1, 2025 14:21

fix lint, use cache for vlm test

29a668c

use caches in other tests

229ef95

sgonorov reviewed Oct 1, 2025

View reviewed changes

sgonorov requested changes Oct 1, 2025

View reviewed changes

create if not existing

dd2140f

akashchi marked this pull request as ready for review October 2, 2025 07:30

Copilot AI review requested due to automatic review settings October 2, 2025 07:30

Copilot AI reviewed Oct 2, 2025

View reviewed changes

tools/who_what_benchmark/tests/constants.py Outdated Show resolved Hide resolved

tools/who_what_benchmark/tests/constants.py Outdated Show resolved Hide resolved

akashchi added 2 commits October 2, 2025 10:39

lint

2bc0ada

use cache for datasets

72ab81a

github-actions bot added category: continuous batching Continuous batching category: whisper Whisper pipeline category: GGUF GGUF file reader labels Oct 2, 2025

as-suvorov reviewed Oct 2, 2025

View reviewed changes

tests/python_tests/test_kv_cache_eviction.py Outdated Show resolved Hide resolved

akashchi added 2 commits October 2, 2025 13:13

Revert "use cache for datasets"

a6028ee

This reverts commit 72ab81a.

verbosity

ffbafd1

github-actions bot added category: GHA CI based on Github actions and removed category: continuous batching Continuous batching category: whisper Whisper pipeline category: GGUF GGUF file reader labels Oct 2, 2025

akashchi added 2 commits October 2, 2025 15:40

use cache for ov models only

57916ca

lint

01a91ed

rm unused

2dcc9ba

akashchi requested a review from Copilot October 6, 2025 12:12

Copilot AI reviewed Oct 6, 2025

View reviewed changes

as-suvorov reviewed Oct 6, 2025

View reviewed changes

tools/who_what_benchmark/tests/constants.py Outdated Show resolved Hide resolved

as-suvorov reviewed Oct 6, 2025

View reviewed changes

tools/who_what_benchmark/tests/constants.py Outdated Show resolved Hide resolved

as-suvorov reviewed Oct 6, 2025

View reviewed changes

tools/who_what_benchmark/tests/test_cli_vlm.py Outdated Show resolved Hide resolved

as-suvorov requested a review from sbalandi October 6, 2025 14:05

as-suvorov mentioned this pull request Oct 7, 2025

[wwb] Add text embeddings pipeline #2787

Merged

3 tasks

quotes, reuse cache dir setting logic

19e5943

akashchi requested a review from as-suvorov October 7, 2025 08:49

as-suvorov mentioned this pull request Oct 7, 2025

[wwb] Add text reranking pipeline #2786

Merged

3 tasks

as-suvorov approved these changes Oct 7, 2025

View reviewed changes

akashchi and others added 2 commits October 7, 2025 14:29

use path to hf model

31bc592

Apply suggestions from code review

f8f7437

Co-authored-by: Alexander Suvorov <[email protected]>

sbalandi approved these changes Oct 7, 2025

View reviewed changes

as-suvorov reviewed Oct 7, 2025

View reviewed changes

tools/who_what_benchmark/tests/test_cli_vlm.py Outdated Show resolved Hide resolved

as-suvorov added 2 commits October 7, 2025 16:59

Update tools/who_what_benchmark/tests/test_cli_vlm.py

42d1eb0

Merge branch 'master' into ci/gha/ov-cache-wwb

e305656

apaniukov reviewed Oct 7, 2025

View reviewed changes

akashchi and others added 6 commits October 8, 2025 09:34

Merge branch 'master' into ci/gha/ov-cache-wwb

0470bf1

Merge branch 'master' into ci/gha/ov-cache-wwb

404461e

rm cache dir

464d2f9

Merge remote-tracking branch 'upstream/master' into ci/gha/ov-cache-wwb

9dfa38b

Merge branch 'ci/gha/ov-cache-wwb' of https://github.com/akashchi/ope…

a8d0882

…nvino.genai into ci/gha/ov-cache-wwb

merge

a6d802a

	MODEL_PATH = MODEL_PATH if MODEL_PATH.exists() else model_id
	MODEL_PATH = str(MODEL_PATH) if MODEL_PATH.exists() else model_id

[CI] [GHA] Use OV_CACHE in the WWB tests #2781

Are you sure you want to change the base?

[CI] [GHA] Use OV_CACHE in the WWB tests #2781

Conversation

akashchi commented Oct 1, 2025

Uh oh!

as-suvorov commented Oct 1, 2025

Uh oh!

sgonorov Oct 1, 2025

Choose a reason for hiding this comment

Uh oh!

sgonorov Oct 1, 2025

Choose a reason for hiding this comment

Uh oh!

sgonorov left a comment

Choose a reason for hiding this comment

Uh oh!

akashchi commented Oct 1, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Copilot AI Oct 6, 2025

Choose a reason for hiding this comment

Uh oh!

akashchi Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 6, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

apaniukov Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

apaniukov Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

apaniukov Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

apaniukov Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

apaniukov Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

as-suvorov commented Oct 7, 2025

Uh oh!

Uh oh!

[CI] [GHA] Use `OV_CACHE` in the WWB tests #2781

[CI] [GHA] Use `OV_CACHE` in the WWB tests #2781

akashchi Oct 6, 2025 •

edited

Loading