Skip to content

Skip MCP integration tests when mcp is not installed or versions predate MCP support#972

Merged
dhalperi merged 3 commits intomasterfrom
copilot/update-mcp-integration-tests
Mar 16, 2026
Merged

Skip MCP integration tests when mcp is not installed or versions predate MCP support#972
dhalperi merged 3 commits intomasterfrom
copilot/update-mcp-integration-tests

Conversation

Copy link
Contributor

Copilot AI commented Mar 16, 2026

  • Add pytest.importorskip("mcp") at module level in tests/integration/test_mcp_server.py to prevent collection failure when mcp is not installed
  • Add autouse fixture _require_versions using skip_old_version to skip tests if pybatfish/batfish version < 2026.3.16
  • Fix ruff import ordering: import pytest (third-party) before import pybatfish (first-party) with blank line separator
  • Verified: collection succeeds with 0 items / 1 skipped when mcp not installed (no error)
  • Verified: ruff check passes
Original prompt

Create a PR in batfish/pybatfish to prevent the new MCP integration tests from failing when the optional mcp dependency is not installed and when cross-version workflows run HEAD tests against older released Pybatfish versions.

Context:

  • A GitHub Actions job in batfish/docker (Cross-Version Tests and Upload) runs integration tests from a pybf_tests artifact (built from HEAD) against various installed Pybatfish versions (e.g., pip install pybatfish[dev]==<old>). This caused pytest collection to fail because tests/integration/test_mcp_server.py imports mcp.server.fastmcp.FastMCP at module import time, producing ModuleNotFoundError: No module named 'mcp'.
  • Pybatfish already has tests/common_util.py with helper requires_bf(version) decorator that skips tests if either Batfish or Pybatfish version is older than a minimum.

Requested change:

  1. Update the new MCP integration test module (tests/integration/test_mcp_server.py, or wherever MCP tests were added) to:

    • Avoid importing mcp at module import time in a way that fails collection when mcp is not installed.
    • Skip the test(s) unless the installed Pybatfish/Batfish version is at least today’s date (YYYY.MM.DD), using the existing @requires_bf(...) decorator.
    • Skip the test(s) if mcp is not installed (use pytest.importorskip('mcp', reason=...) or equivalent). Ensure the skip happens before any mcp imports.
  2. Ensure any version string used matches the project’s version format expectations (e.g., YYYY.MM.DD or YYYY.MM.DD.####). The user requested using today’s date (current date: 2026-03-16) as the minimum version, so use 2026.03.16 (or include a patch segment if needed by existing logic) consistently.

  3. Make the change minimal and targeted:

    • Prefer only changes in test code; do not modify packaging unless necessary.
    • Ensure pytest collection succeeds even without mcp installed.

Acceptance criteria:

  • Running pytest tests/integration in an environment without mcp installed does not error during collection; MCP-related tests are skipped with a clear reason.
  • Running the MCP tests in an environment where mcp is installed and versions are >= 2026.03.16 proceeds normally.
  • Cross-version workflow scenario (HEAD tests tarball used with older Pybatfish installs) no longer fails due to missing mcp.

Repository: batfish/pybatfish

The following is the prior conversation context from the user's chat exploration (may be truncated):

User: We just added the mcp support to the head development version of pybatfish. Why would this failing test be running on our old version, and what do we need to do to fix it? It also seems odd that it's running integration tests from the head version of pybf instead of the version it installed?

Assistant: This failure is happening because the cross-version workflow in batfish/docker deliberately runs “HEAD” integration tests against released/older Pybatfish installs—and your new MCP integration test now imports mcp, which isn’t present in those older releases’ dependency sets.

Why is it running on an “old version” of pybatfish?

In batfish/docker’s reusable workflow, the pybf_cross_version_tests job installs Pybatfish from PyPI at specific historical versions:

- name: Setup pybf
  run: pip install "pybatfish[dev]==${{ matrix.pybf_version }}" "urllib3<2"

...
- name: Run tests
  run: |
    TEMP_DIR=$(mktemp -d)
    tar xzf pybatfish-tests.tgz -C ${TEMP_DIR}
    ...
    pytest -k "not test_notebook_output" "${TEMP_DIR}/tests/integration"

So it’s expected to be running tests while pybatfish==<old-version> is installed.

Why does it seem to run HEAD integration tests instead of the installed version?

Also by design: the workflow downloads the pybf_tests artifact (which comes from the HEAD build) and runs those tests regardless of which Pybatfish version is installed:

- name: Download pybf tests
  uses: actions/download-artifact@v8
  with:
    name: pybf_tests
...
pytest -k "not test_notebook_output" "${TEMP_DIR}/tests/integration"

That’s why you’re seeing tests/integration/test_mcp_server.py from HEAD being collected, even though the installed package is an older pybatfish.

What exactly is failing?

Your failing job log shows test collection fails because mcp isn’t installed:

tests/integration/test_mcp_server.py:31: in <module>
  from mcp.server.fastmcp import FastMCP
E   ModuleNotFoundError: No module named 'mcp'

What you need to do to fix it

You have two viable options (pick base...

This pull request was created from Copilot chat.


📱 Kick off Copilot coding agent tasks wherever you are with GitHub Mobile, available on iOS and Android.

@batfish-bot
Copy link

This change is Reviewable

… old

Co-authored-by: dhalperi <526415+dhalperi@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix MCP integration tests for optional dependency Skip MCP integration tests when mcp is not installed or versions predate MCP support Mar 16, 2026
Copilot AI requested a review from dhalperi March 16, 2026 20:39
Copy link
Member

@dhalperi dhalperi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dhalperi reviewed 1 file and all commit messages.
Reviewable status: :shipit: complete! all files reviewed, all discussions resolved (waiting on copilot[bot]).

@dhalperi dhalperi marked this pull request as ready for review March 16, 2026 20:41
@dhalperi dhalperi enabled auto-merge (squash) March 16, 2026 20:41
@dhalperi
Copy link
Member

@copilot fix failures.

…arty

Co-authored-by: dhalperi <526415+dhalperi@users.noreply.github.com>
auto-merge was automatically disabled March 16, 2026 20:45

Head branch was pushed to by a user without write access

Copy link
Contributor Author

Copilot AI commented Mar 16, 2026

@copilot fix failures.

Fixed in 618ea05ruff check flagged an import ordering issue: import pybatfish (first-party) needed to come after import pytest (third-party), separated by a blank line.

Copilot AI requested a review from dhalperi March 16, 2026 20:46
Copy link
Member

@dhalperi dhalperi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dhalperi reviewed 1 file and all commit messages.
Reviewable status: :shipit: complete! all files reviewed, all discussions resolved (waiting on copilot[bot]).

@dhalperi dhalperi enabled auto-merge (squash) March 16, 2026 20:47
@dhalperi dhalperi merged commit 2c76d06 into master Mar 16, 2026
17 checks passed
@dhalperi dhalperi deleted the copilot/update-mcp-integration-tests branch March 16, 2026 20:49
@codecov
Copy link

codecov bot commented Mar 16, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.77%. Comparing base (cdec025) to head (618ea05).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #972   +/-   ##
=======================================
  Coverage   87.77%   87.77%           
=======================================
  Files          26       26           
  Lines        3763     3763           
  Branches      501      501           
=======================================
  Hits         3303     3303           
  Misses        330      330           
  Partials      130      130           
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants