Karate test results no longer included in the overall report with update of maven surefire plugin version

[JoanaPedrosoDiconium]

I updated the maven surefire plugin version from 3.5.2 to 3.5.3 and since then the test results of the Karate Tests no longer appear in the overall test report, even though the tests run.

This is the configuration we have (and the one we had before the version update):

<dependencies>
        <dependency>
            <groupId>io.karatelabs</groupId>
            <artifactId>karate-junit5</artifactId>
            <scope>test</scope>
            <version>${karate-junit5.version}</version>
        </dependency>
</dependencies>
<build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <configuration>
                    <layers>
                        <enabled>true</enabled>
                    </layers>
                    <excludes>
                        <exclude>
                            <groupId>org.projectlombok</groupId>
                            <artifactId>lombok</artifactId>
                        </exclude>
                    </excludes>
                </configuration>
                <executions>
                    <execution>
                        <goals>
                            <goal>build-info</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
           <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-report-plugin</artifactId>
                <version>${maven-surefire-plugin.version}</version>
            </plugin>
</build>

In terms of folder structure, we have the entry point of the KarateTests in the src/test/java folder but the feature files are in the src/test/resources/karate-tests folder and we would like to keep this structure. Like I said with this setup and configurations we had no problems before.

Is there any hints you can give us to be able to update to the new version of the plugin and keep the Karate Tests results in our overall report?

Thank you!

[ptrthomas]

most likely a library conflict. the best reference for you to troubleshoot is this: #2436
we will not be able to do anything here unless you follow this process: https://github.com/karatelabs/karate/wiki/How-to-Submit-an-Issue

[JoanaPedrosoDiconium]

I tried to replicate the issue using that karate-template that you linked but even with the maven surefire plugin version 3.5.2 I was not getting the tests results of the karate tests. In any case test structure is different in the template comparing with my project. The solution I found was to make karate test cases run with maven failsafe plugin whilst the rest of my test cases run with the surefire. Like this, my target folder gets the karate-reports subfolder as well as the surefire-reports and failsafe-reports folders that contain xml files with the test results:

I am using dorny as the test reporter on my Github actions and had to change a bit the workflow files it uses as well as the maven phase it runs, but now I am able to include all tests results including karate, in the final result.

I still do not understand what changed on the surefire side. But to be fair, this is probably a better approach as it is said in the documentation that failsafe should be used for integration tests and surefire for unit tests and Karate Tests are imo not unit tests.