Skip to content

Skip newer architecture config tests on older transformers versions#1223

Open
brendanlong wants to merge 1 commit intoTransformerLensOrg:devfrom
brendanlong:brendanlong/old-transformers-tests
Open

Skip newer architecture config tests on older transformers versions#1223
brendanlong wants to merge 1 commit intoTransformerLensOrg:devfrom
brendanlong:brendanlong/old-transformers-tests

Conversation

@brendanlong
Copy link
Copy Markdown

Description

The Apertus and GPT-OSS architecture tests fail when the installed transformers version doesn't recognize these model types. Skip with a clear message instead of failing. Also moves inline imports to top-level.

This allows us to run all of the unit tests on transformers 4.42, which is supported on Python 3.8:

https://github.com/TransformerLensOrg/TransformerLens/blob/main/pyproject.toml#L39

Type of change

  • Bug fix (non-breaking change which fixes an issue)

Screenshots

With poetry run pip install transformers==4.42.0.

Before:

> poetry run pytest tests/unit/test_loading_from_pretrained_utilities.py 
========================================================= test session starts =========================================================
platform linux -- Python 3.12.9, pytest-8.3.5, pluggy-1.5.0
rootdir: /home/brendanlong/workspace/TransformerLens
configfile: pyproject.toml
plugins: jaxtyping-0.2.19, doctestplus-1.3.0, nbval-0.10.0, cov-5.0.0, anyio-4.5.2, typeguard-4.4.0
collected 8 items                                                                                                                     

tests/unit/test_loading_from_pretrained_utilities.py .....FFF                                                                   [100%]

============================================================== FAILURES ===============================================================
_____________________________________________ TestArchitectureConfigs.test_apertus_config _____________________________________________

[...]
../../.cache/pypoetry/virtualenvs/transformer-lens-MODGY9P9-py3.12/lib/python3.12/site-packages/transformers/models/auto/configuration_auto.py:982: 
[...]
../../.cache/pypoetry/virtualenvs/transformer-lens-MODGY9P9-py3.12/lib/python3.12/site-packages/transformers/models/auto/configuration_auto.py:684: KeyError

During handling of the above exception, another exception occurred:
[...]
tests/unit/test_loading_from_pretrained_utilities.py:102: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../.cache/pypoetry/virtualenvs/transformer-lens-MODGY9P9-py3.12/lib/python3.12/site-packages/jaxtyping/decorator.py:121: in wrapped_fn
    return fn(*args, **kwargs)
<@beartype(transformer_lens.loading_from_pretrained.get_pretrained_model_config) at 0x7988818c3ec0>:247: in get_pretrained_model_config
    ???
transformer_lens/loading_from_pretrained.py:2228: in get_pretrained_model_config
    cfg_dict = convert_hf_model_config(official_model_name, **kwargs)
../../.cache/pypoetry/virtualenvs/transformer-lens-MODGY9P9-py3.12/lib/python3.12/site-packages/jaxtyping/decorator.py:121: in wrapped_fn
    return fn(*args, **kwargs)
<@beartype(transformer_lens.loading_from_pretrained.convert_hf_model_config) at 0x7988818a2b60>:31: in convert_hf_model_config
    ???
transformer_lens/loading_from_pretrained.py:862: in convert_hf_model_config
    hf_config = AutoConfig.from_pretrained(
[...]
E               ValueError: The checkpoint you are trying to load has model type `apertus` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.

../../.cache/pypoetry/virtualenvs/transformer-lens-MODGY9P9-py3.12/lib/python3.12/site-packages/transformers/models/auto/configuration_auto.py:984: ValueError
[etc.]
FAILED tests/unit/test_loading_from_pretrained_utilities.py::TestArchitectureConfigs::test_apertus_config - ValueError: The checkpoint you are trying to load has model type `apertus` but Transformers does not recognize this architecture. ...
FAILED tests/unit/test_loading_from_pretrained_utilities.py::TestArchitectureConfigs::test_gpt_oss_config - ValueError: The checkpoint you are trying to load has model type `gpt_oss` but Transformers does not recognize this architecture. ...
FAILED tests/unit/test_loading_from_pretrained_utilities.py::TestArchitectureConfigs::test_apertus_instruct_config - ValueError: The checkpoint you are trying to load has model type `apertus` but Transformers does not recognize this architecture. ...
============================================== 3 failed, 5 passed, 53 warnings in 7.85s ===============================================

After:

> poetry run pytest tests/unit/test_loading_from_pretrained_utilities.py 
========================================================= test session starts =========================================================
platform linux -- Python 3.12.9, pytest-8.3.5, pluggy-1.5.0
rootdir: /home/brendanlong/workspace/TransformerLens
configfile: pyproject.toml
plugins: jaxtyping-0.2.19, doctestplus-1.3.0, nbval-0.10.0, cov-5.0.0, anyio-4.5.2, typeguard-4.4.0
collected 8 items                                                                                                                     

tests/unit/test_loading_from_pretrained_utilities.py .....sss                                                                   [100%]

========================================================== warnings summary ===========================================================
../../.cache/pypoetry/virtualenvs/transformer-lens-MODGY9P9-py3.12/lib/python3.12/site-packages/jaxtyping/import_hook.py:106: 52 warnings
  /home/brendanlong/.cache/pypoetry/virtualenvs/transformer-lens-MODGY9P9-py3.12/lib/python3.12/site-packages/jaxtyping/import_hook.py:106: DeprecationWarning: ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead
    elif isinstance(child, ast.Expr) and isinstance(child.value, ast.Str):

transformer_lens/SVDInterpreter.py:26
  /home/brendanlong/workspace/TransformerLens/transformer_lens/SVDInterpreter.py:26: InstrumentationWarning: @typechecked only supports instrumenting functions wrapped with @classmethod, @staticmethod or @property -- not typechecking transformer_lens.SVDInterpreter.SVDInterpreter.get_singular_vectors
    @typechecked

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
============================================== 5 passed, 3 skipped, 53 warnings in 7.45s ==============================================

Checklist:

  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I have not rewritten tests relating to key interfaces which would affect backward compatibility

@brendanlong brendanlong changed the base branch from main to dev March 29, 2026 16:35
The Apertus and GPT-OSS architecture tests fail when the installed
transformers version doesn't recognize these model types. Skip with
a clear message instead of failing. Also moves inline imports to
top-level.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@brendanlong brendanlong force-pushed the brendanlong/old-transformers-tests branch from 4fb5bc6 to 705436b Compare March 29, 2026 17:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant