Citrus 2.7.2 test cases can run inside citrus remote container but citrus remote server always get empty test results

[KennethCool]

Hi Dear,
I have legacy test cases of:

• citrus 2.7.2,
• test cases are of junit 4.13
• spring-core: 4.3.2.RELEASE
• spring-context: 4.3.2.RELEASE
• running in jre1.8

Now I tried to run those test cases in a container by citrus-remote maven plugin. Test cases can be found and run successfully.
But the testing result returned to citrus-remote maven plugin of test is always 0 test cases:

INFO] --- citrus-remote:4.6.0:test (trigger-citrus-remote) @ <my_submodule name> ---
[INFO] 
INFO	| 
INFO	| CITRUS TEST RESULTS
INFO	| 
INFO	| 
INFO	| TOTAL:		0
INFO	| PASSED:		0 (0.0%)
INFO	| FAILED:		0 (0.0%)
INFO	| TIME:		0 ms
INFO	| 
INFO	| ------------------------------------------------------------------------

[INFO] Generated test report: <my_project_dir>/test/<my_submodule>/target/citrus-remote/citrus-test-results.html
[INFO] Generated test report: <my_project_dir>/test/<my_submodule>/target/citrus-remote/citrus-summary.xml
[INFO] Writing report file: <my_project_dir>/test/<my_submodule>/target/citrus-remote/TEST-TestSuite.xml

The result is the same whether I run test cases in my docker container, or run by 'java -jar citrus-tests.jar --duration 9223372036854775807 ' directly in my host.

The environment of running citrus-remote test jar in my container

• citrus-remote maven plugin: 4.5.2, or latest 4.6.0, the same.
• jdk17 (should not cause trouble, since all the test cases can finish successfully)
• Other dependencies of legacy citrus 2.7.x test cases are the same.

Reproduce

A simplest test case is tried and reproduce the issue:

import com.consol.citrus.annotations.CitrusTest;
import com.consol.citrus.dsl.junit.JUnit4CitrusTest;
import org.junit.Test;

public class DummyTest extends JUnit4CitrusTest {
    
    @Test
    @CitrusTest
    public void kenTest() {
        System.out.println("DummyTest#kenTest");
    }
}

By running via jar directly (the same result if running inside docker container)

=============================
java -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005 --add-opens java.base/java.lang=ALL-UNNAMED  -jar test/<my_module>/target/<my_module>-50.0-feature-dummy-citus.0-SNAPSHOT-citrus-tests.jar --duration 9223372036854775807 --skipTests true
=============================

Listening for transport dt_socket at address: 5005
00:03:12,641 INFO  rusRemoteApplication| Server started on port 4567
00:03:21,976 INFO  rusRemoteApplication| PUT /run
00:03:22,071 INFO  rusRemoteApplication| GET /results?timeout=999999000
00:03:22,234 INFO        main.CitrusApp| Running Citrus 4.6.0
00:03:22,444 INFO  n.JarFileTestScanner| Found test class candidate in test jar file: com/<my_company>/citrus/<sub_dir>/DummyTest.class
00:03:22,447 INFO  nit.JUnit4TestEngine| Found 1 test classes to execute
00:03:22,452 INFO  nit.JUnit4TestEngine| Running test com.my_company.citrus.inquiry.paymentstatus.mf.DummyTest
DummyTest#kenTest
00:03:22,556 INFO  rusRemoteApplication| GET /results/files
00:03:22,571 INFO  rusRemoteApplication| GET /results/suite

Analysis

As I debug the citrus-remote server side, I found results in RemoteTestListener is never updated, including by onTestSuccess. So results is always empty.
And I saw in CitrusRemoteApplication, the remoteTestListener is added as:

CitrusInstanceManager
                .addInstanceProcessor(citrus -> citrus.addTestListener(remoteTestListener));

To trigger citrus testing run by citrus remote maven plugin

<plugins>
					<plugin>
						<groupId>org.citrusframework</groupId>
						<artifactId>citrus-remote-maven-plugin</artifactId>
						<version>${citrus-remote-maven-plugin}</version>
						<executions>
							<execution>
								<id>trigger-citrus-remote</id>
								<goals>
									<goal>test</goal>
								</goals>
								<phase>${citrus.remote.test.phase}</phase>
								<configuration>
									<server>
										<url>http://localhost:4567</url>
									</server>
									<run>
										<engine>junit4</engine>
										<async>true</async>
										<pollingInterval>999999000</pollingInterval>
										<classes>
										</classes>
										<includes>
											    <include>^.*DummyTest$</include>
										</includes>
										<systemProperties>
											<property>
												<name>file.encoding</name>
												<value>UTF-8</value>
											</property>
											<property>
												<name>citrus.file.encoding</name>
												<value>UTF-8</value>
											</property>
										</systemProperties>
									</run>
									<report>
										<directory>${citrus.remote.report.directory}</directory>
									</report>
								</configuration>
							</execution>

Citrus-remote-sample

I tried Citrus-remote-sample, which works fine and can show 3 successful test cases in results. All good.
One thing is the demo GetTextIT is TestNG, not junit4.

I tried citrus 2.7.2, 2.7.9, got same issue.
Is there any way that onTestSuccess can be triggered so that results can be updated?
Any configurations of citrus 2.7.x, or of citrus remote plugin can help? Thanks

[KennethCool]

My legacy test cases dependencies:

• citrus-core: 2.7.2
• citrus-java-dsl: 2.7.2
• citrus-http: 2.7.2
• citrus-jms: 2.7.2
• citrus-docker: 2.7.2

E.g.,

groupId>com.consol.citrus</groupId>
			<artifactId>citrus-core</artifactId>
			<version>${citrus.version}</version>

[KennethCool]

I tried running tens of legacy test cases, all of which can be found and run successfully when I test against the citrus-remote.jar directly. After the last test case was successful, a result is gathered by citrus-remote-server module, however, it is empty, same as that I tried with DummyTest above.
But logs at the citrus-remote-server side, show that all the test cases run successfully. E.g.,

00:39:17,840 INFO         citrus.Citrus| 
00:39:17,840 DEBUG        citrus.Citrus| TEST STEP 39 SUCCESS
00:39:17,841 INFO         citrus.Citrus| 
00:39:17,841 INFO         citrus.Citrus| TEST SUCCESS Status Inquiry with all field supplied (com.<my_company>.citrus.<my_package>)

These logs are from legacy citrus 2.7.x

[KennethCool]

As a comparison, in the module of citrus-remote-sample, I added a juni4 dummy test case. The citrus-remote server can gather testing result correctly.
So junit4 test case can work if on citrus version 4.6.x, while my legacy test cases are on citrus 2.7.x.

import org.citrusframework.annotations.CitrusTest;
import org.junit.Test;

import org.citrusframework.junit.JUnit4CitrusSupport;

public class Simple_IT extends JUnit4CitrusSupport {

    @Test
    @CitrusTest(name = "Simple_IT")
    public void simpleTest() {
        System.out.println("Simple_IT dummy one");
    }
}

[INFO] --- citrus-remote:4.7.0-SNAPSHOT:test (trigger-citrus-remote) @ citrus-remote-sample ---
[INFO] Waiting for remote tests to finish ...
[INFO] 
[INFO] Waiting for remote tests to finish ...
[INFO] 
[INFO] Waiting for remote tests to finish ...
[INFO] 
[INFO] Waiting for remote tests to finish ...
[INFO] 
[INFO] Waiting for remote tests to finish ...
[INFO]
[INFO] 
INFO	| 
INFO	| CITRUS TEST RESULTS
INFO	| 
INFO	| SUCCESS (   582 ms) Simple_IT
INFO	| 
INFO	| TOTAL:		1
INFO	| PASSED:		1 (100.0%)
INFO	| FAILED:		0 (0.0%)
INFO	| TIME:		582 ms
INFO	| 
INFO	| ------------------------------------------------------------------------

Junit ver: 4.13

[KennethCool]

If I use citrus remote plugin and remote server modules of citrus version 2.8.0, instead of version 4.x, will it work with my legacy test cases of citrus 2.7.2? Thanks.

Code analysis

Found differences between citrus versions 4.x and 2.7.2, which cause the testing result can or cannot be collected by citrus remote server:

Set remote test listeners

Citrus-remote on version 4.x:

• CitrusContext class stores citrus context
• Citrus class has addTestListener(), which allows the remoteTestListener of citrus-remote server to be added. The remoteTestListener will then be added to CitrusContext class

Then when Loading remote test listener

• If test cases are based on Version 4.x:

In JUnit4CitrusSupport.java

Load citrus context by citrus.getCitrusContext(), then from which further get the test listener

• My test cases on version 2.7.2, which extends JUnit4CitrusTest. But JUnit4CitrusTestcannot load citrus context

[KennethCool]

Seems the citrus-remote module of citrus 2.8.0 cannot help either:

Though the citrus remote server adds the test listener by citrus.addTestListener(remoteTestResultReporter), but it cannot be loaded by the test cases that extends JUnit4CitrusTest.

[KennethCool]

Find one solution. Could you help to check if it has any problem? I tested it and it worked with my test case of citrus 2.7.x. The citrus remote server of ver4.6 can gather testing results as expected.

The solution is to
Add custom class extends AbstractTestListener. It:

• Loads RemoteTestListener from Citrus of version 4.x, when a test starts
• Does the similar logic as RemoteTestListener does when onTestSuccess

import org.citrusframework.Citrus;
import org.citrusframework.TestResult;
import org.citrusframework.remote.listener.RemoteTestListener;
import org.citrusframework.report.TestListener;
import org.citrusframework.report.TestListeners;
import org.citrusframework.report.TestResults;

public class CitrusRemoteTestListener extends AbstractTestListener implements InitializingBean {

private static RemoteTestListener remoteTestListener;

private final Map<ClassAndName, Long> startTimes = new HashMap<>();

private void loadRemoteTestListener() {
        if (remoteTestListener == null) {
            Citrus citrus = Citrus.newInstance();
            if (citrus.getCitrusContext() == null) {
                LOG.error("Cannot get CitrusContext from Citrus instance");
                return;
            }
            TestListeners testListeners = citrus.getCitrusContext().getTestListeners();
            if (testListeners == null) {
                LOG.error("Cannot get TestListeners from Citrus instance");
                return;
            }
            boolean found = false;
            for (TestListener testListener : testListeners.getTestListeners()) {
                if (testListener instanceof RemoteTestListener) {
                    LOG.info("Found RemoteTestListener");
                    remoteTestListener = (RemoteTestListener) testListener;
                    found = true;
                    break;
                }
            }
            if (!found) {
                LOG.error("Cannot find RemoteTestListener");
            }
        }
    }

    @Override
    public void onTestStart(TestCase test) {
        loadRemoteTestListener();
        startTimes.put(ClassAndName.of(test), System.currentTimeMillis());
    }

    @Override
    public void onTestSuccess(TestCase test) {
        // Cannot call remoteTestListener.onTestSuccess(test), because its input arg type is expected to be org.citrusframework, which is of 
        // a new version used by citrus remote module, is different to com.consol.citrus.TestCase used by citrus 2.7.x
        TestResults testResults = remoteTestListener.getResults();
        testResults.addResult(TestResult.success(
                        test.getName(),
                        test.getTestClass().getCanonicalName(),
                        test.getVariableDefinitions())
                .withDuration(getDuration(test)));
    }
}