Line coverage generated through batchmode different than line coverage when using editor

Unity Engine
, ,
1

Hello! We run code coverage through the command line for our build automation. The line coverage generated when we run code coverage in batch mode is different than when run through the editor. We’ve observed this behaviour on all of our projects. It seems like when run through batchmode, code coverage includes assemblies that are not included in the project’s Settings.json file, even though the useProjectSettings option is enabled.

We are generating a single report for both edit mode and play mode tests. Here are the commands we use:
(We use the same command for playmode, but with the testPlatform set to PlayMode)

Unity.exe
-runTests
-batchmode
-buildTarget WebGL
-testPlatform EditMode
-projectPath .
-logFile "<path-here>"
-testResults "<path-here>"
-debugCodeOptimization
-enableCodeCoverage
-coverageResultsPath "<path-here>"
-coverageOptions "generateAdditionalMetrics;useProjectSettings;dontClear"

Unity.exe
-batchmode
-buildTarget WebGL
-logFile "<path-here>"
-projectPath .
-debugCodeOptimization
-enableCodeCoverage
-coverageResultsPath "<path-here"
-coverageOptions "generateHtmlReport;generateHtmlReportHistory;generateBadgeReport;generateAdditionalMetrics;useProjectSettings"
-quit

This is run through Jenkins.

2

Hi @lbunting .

Do you get the same outcome when you -runTests and produce the html report in the same command i.e. just for EditMode?

Could you please report a bug with a small repro project? That way we will be able to investigate further.

Thank you.

3

Also try removing the dontClear option

4

Thanks for the speedy reply, @Unity-Nikos . :slight_smile:

Using -runTests and producing the html report in the same command for 1 type of test works as intended, it seems like it happens only when I generate a single report for both. I tried removing the dontClear option, but that did not affect the outcome.

I’ll put together a repro project and submit a bug report ASAP.

5

Sorry for this taking so long, I had quite a few other things on my plate. @Unity-Nikos , the bug report has been submitted. :slight_smile:

1 Like