Code Coverage Package - Discussion

The Code Coverage package has now been released. You can find the new introduction thread here: Code Coverage Package - Release Announcements - Unity Engine - Unity Discussions


Use the Code Coverage preview package with the Test Runner to gather and present test coverage information. When you run your tests with code coverage enabled, you can see exactly which lines of your code are executed, in addition to whether the tests have passed or failed. For more details see Using Code Coverage with Test Runner.


Code Coverage window

Once a test run has completed, the Code Coverage package will generate an HTML coverage report showing which lines of your code are covered by tests. Code Coverage currently supports EditMode and PlayMode tests in the Editor. It also allows you to track the code coverage changes through time.


HTML report

If you don’t have any tests, don’t worry! The Code Coverage package offers a Coverage Recording feature which allows capturing coverage data on demand.

5174774--524954--coverage-recording.gif

The Code Coverage package is available as a preview package via the Package Manager for Unity 2019.3 and above. See Installing Code Coverage. Please make sure to enable preview packages in the Package Manager, otherwise it won’t be listed.

Take note that preview packages are not considered ready for production use and might change before being verified for release.


Installing Code Coverage

For more information see the Code Coverage package documentation.

We look forward to your questions and feedback!

16 Likes

I just tried it out and just gotta say, wow, this is an awesome tool to have at your disposal! I try to work with TDD where possible so I thought I had 100% test coverage in some of my submodules and with the help of this I immediately spotted som holes that could have meant bugs could creep in.

I really like that I can see the history too :slight_smile:

It’s nice that I can place the report in a folder of my choice as I want to test submodules separately and store the report in each submodule. Would it be possible to add some concept of “presets” so that I can define one test report for each submodule and then easily switch between them? As it is right now I would need to do this manually and must remember wehere each submodule stores its report.

Also, it warns for poor performance when being turned on, is this only when you generate the report or does it have some kind of constant overhead?

I’ll try it out more and get back to you with more questions probably :slight_smile:

4 Likes

Hi Mikael,

thanks for the feedback and suggestions!

Presets:
This is a good idea and agree that having separate presets for separate reports would give more flexibility. Currently the results are saved under a <ProjectName>-opencov folder, so with presets this could be updated to <ProjectName>-<PresetName>-opencov. What we would not be able to do is to reflect the presets in the report (i.e. a single report with different presets)…for now at least. I will discuss it with the team and add it to the backlog, and post any updates here.

Code Coverage performance overhead:
When you check Enable Code Coverage in Preferences > General you enable access to the interface for the code coverage data that Mono exposes - see Unity - Scripting API: Coverage and Technical details | Code Coverage | 0.2.3-preview. This adds some overhead to the editor and lowers the performance, so it is not recommended to leave it on if you are not doing coverage testing. We are exploring changing this in the future. We will post any progress here.

Thanks
Nikos

3 Likes

Thanks for the clarification! I’ll turn it off when not actively TDD:ing then :slight_smile:

Regarding presets, I’m sure whatever you guys come up with will be awesome. I don’t think it is a super necessary feature, just nice to have.

1 Like

Is possible to use it in unity 2017?

Answer:

Unfortunately not as the Code Coverage API was introduced in Unity 2019.2

Hello, I used code coverage tool in Unity 2020.1 today and I want to use this tool for a long time. However, I have encountered several problems. I don’t know if it’s my own operation problem or a tool bug.

I wrote a demo to learn to use code coverage tool, which is a class including function add, sub, mul and div and a Print function:


Then I wrote a test class in playmode, including the test function of function add, sub, mul and div(not including print function).

Then I ran the playmode test and generated the code coverage,which is out of my expectation:

The functions which are tested by the test function are not covered and the print function which is not tested by the test function is covered!

Then I found out that the reason why the Print function is covered is that I invoked that function in Start function. I can’t understand why this is handled like this: Shouldn’t coverage be the part of the code being tested? The invoke of Print function is not in the test code.

There is a second question. I added some log infomation in the add, sub, mul and div functions:


Then I ran the test, the result surprised me again:

The coverage is 100% now! I did nothing but add log statement in the class! I can’t understand the result like this. Can you troubleshoot the answer? Thank you very much!! Looking forward to your reply!!

(The website doesn’t allow me to upload source code so I have to use picture instead, forgive me.)

Hi DavidWei1. I suspect it is due to Code optimization been set to Release mode. Code Optimization was introduced in 2020.1. Code Optimization mode defines whether Unity Editor compiles scripts in Debug or Release mode. Debug mode enables C# debugging and it is required in order to obtain accurate code coverage. To ensure Code optimization is set to Debug mode you can do one of the following:

  • Switch to Debug mode in the Editor (bottom right corner, select the Bug icon > Switch to debug mode)
  • Using the CompilationPipeline api, set CompilationPipeline.codeOptimization = CodeOptimization.Debug
  • Pass -debugCodeOptimization to the command line

Please see Using Code Coverage | Code Coverage | 0.2.3-preview

Let me know if this helps. We are planning to do this automatically in the future.

Thanks
Nikos

1 Like

H Aaron, the package is available in Unity 2019.3 and above. We could not support 2017 as the coverage API was added in Unity 2019 - please see https://docs.unity3d.com/Packages/com.unity.testtools.codecoverage@0.2/manual/TechnicalDetails.html

1 Like

It works!! The coverage is 100% whether I have added logs or not.
However, I still have a question. The meaning of coverage here is the coverage of the test code or the full coverage of all the code runs? As I didn’t write the test code of Print function but I invoked that function in Start function, the coverage increased still. I am a bit confused.
Once upon a time I used googletest to get the coverage of my C++ unit test, its coverage is only about the code I covered in my test code.
Looking forward to your reply!!! Thank you very much!!

@ since Class1 is a MonoBehaviour, depending on how you initialise it in your playmode test (I cannot see that bit in the screenshots), I suspect that Start is triggered as the script gets added to the Scene, or perhaps it is already added in the Scene before you run your test

1 Like

@Unity-Nikos ok, so here’s another slightly dreamy feature request :slight_smile: Would it be possible to somehow use the test coverage report with SonarQube? That would really make this fit in well at work where we use SonarQube for static code analysis and over time test coverage measuring on all builds. Not sure how many would be interested in this but I suspect many in the industrial segment where Unity is part of a larger ecosystem would be…

Hi, Nikos. I get a new question. I want to use batchmode to run unit test and get coverage report. How can I change Unity to debug mode using command line?

I changed to use Unity2019.3.0b12, which doesn’t have the little bug at its right bottom conner and the problem is solved!

@DavidWei1 in 2020.1 you can pass -debugCodeOptimization to the command line.

See my reply above Code Coverage Package - Discussion

1 Like

@Mikael-H try pointing SonarQube to the <project-name>-opencov folder located under CodeCoverage in the project’s root folder or under the Destination folder.

1 Like

Hi Nikos, I have a new problem here. Can Code Coverage only be used to determine the coverage of which part of the code to test? There is a part of code in my project that is automatically generated and does not require test coverage, but if I simply specify Assembly-CSharp to include they will be included in my coverage as well. Is there a way to solve it? Looking forward to your reply!

Hi @DavidWei1 currently you could achieve this by adding amsdefs to organise your code, thus be able to filter by those new assemblies. See Unity - Manual: Assembly definitions

We are also adding a command line feature where you could filter out directories, in which case you could filter out the directory where the code is auto-generated into.

Thanks
Nikos

May I ask if the using of the command line to filter out directory is available now?

It will be in the next version of the package, 0.2.1-preview. We will post an update to this thread when this is live.