[BUG] Possible mismatch in pre-run modifier handling between RobotCode and Robot Framework CLI

[simonmeggle]

Hi Daniel, I noticed when running Robot Framework tests via RobotCode using a profile that includes a pre-run modifier, dynamically added tests are not recognized:
Robot Framework reports that the suite contains no tests after model modifiers.

Running the exact same setup via the Robot Framework CLI works correctly and executes the dynamically created tests as expected.

I hope I am not wrong, but there seems to be a difference in how RobotCode invokes Robot Framework or applies pre-run modifiers compared to the CLI.

Steps To Reproduce

Create a Robot Framework test file with a single dummy test:

*** Test Cases ***
Dummy
    No Operation

And a pre-run modifier (MyPrerunModifier.py):

from robot.api import SuiteVisitor

# This prerun modifier creates 3 dynamic tests based on a single "template" test.
# The original test is removed from the suite.

class MyPrerunModifier(SuiteVisitor):
    def start_suite(self, suite):
        tests = []
        if suite.tests:
            for i in range(1, 4):
                new_test = suite.tests[0].deepcopy()
                new_test.name = f"test number {i}"
                tests.append(new_test)
            suite.tests.clear()
            suite.tests.extend(tests)

robot.toml:

[profiles.PrerunModifier]
[profiles.PrerunModifier.pre-run-modifiers]
"./MyPrerunModifier.py" = []

After that, run the test using RobotCode with the PrerunModifier profile.

Expected behavior
Robot Framework should execute these dynamically generated tests:
• test number 1
• test number 2
• test number 3

This is exactly what happens when running Robot Framework manually via the CLI using the same pre-run modifier:

robot --prerunmodifier MyPrerunModifier.py  tests_prm.robot
==============================================================================
Tests Prm                                                                     
==============================================================================
test number 1                                                         | PASS |
------------------------------------------------------------------------------
test number 2                                                         | PASS |
------------------------------------------------------------------------------
test number 3                                                         | PASS |
------------------------------------------------------------------------------
Tests Prm                                                             | PASS |
3 tests, 3 passed, 0 failed

Actual behavior when run with Robotcode

[ ERROR ] Suite 'Rf-Listener Create Dynamic Tests' contains no tests after model modifiers.

Environment

• VS Code Version 1.108.2
• RobotCode Version 2.2.0
• OS: Mac, Windows
• Python Version 3.12.3
• RobotFramework Version 7.1.1