Test Runner EditorMode / PlayMode tests and code coverage with NUnit

Unity Engine
1

Hi,

As you some know (like @QA-for-life @ElvisAlistar @ ) I’m building our continuous integration system that can build projects (Jenkins) and perform code analysis and coverage (Sonar). Sonarqube And Unity (code Quality) - Unity Engine - Unity Discussions

I was looking after PlayMode tests that didn’t made it into 5.5.

I’m now using it in 5.6 beta and works really great.

I see that you now use NUnit 3 (3.6).

I also want to get the code coverage results but in “command line arguments” page there is no command available for code coverage (only for unit test execution with -runEditorTests): Unity - Manual: Command-line arguments

I can’t do it myself from outside editor because: “System.Security.SecurityException : ECall methods must be packaged into a system module.”

Now I’m trying to do it with:

Can you add it please? It shouldn’t be difficult.

3 Likes
2

Bump.

Can you answer please? @QA-for-life @ElvisAlistar @

3

Bump.

I insist @QA-for-life @ElvisAlistar @ . It’s a really important matter. Unit testing without code coverage it’s useless.

I’ve in our pipeline, nightly builds from Jenkins that triggers a build step that runs the code analysis and coverage that reports it to SonarQube but I can’t run tests that use Unity code because of this.

Can I have at least an answer? Can I chat with any of you, please? Thanks.

4

Code coverage is a just a metric like many other and unit tests have a great value by themselfs. We don’t have a code coverage tooling in the plans for the nearest future.

5

Is there no workaround for this @Tomek-Paszek ? It would be really useful if we could run a code coverage analysis by ourselves.

Now we only can a code coverage analysis on code that don’t use Unity API (System.Security.SecurityException : ECall methods must be packaged into a system module.).

6

Some time ago we evaluated few open source solutions but haven’t found anything that would work with Unity code out of the box. We haven’t tried commercial solutions.

7

Can’t you at least enable it with the solution I proposed (Visual Studio it’s used by almost anyone)? Code coverage is something advanced that it’s not going to be used by people starting to learn unity, it’s mostly used by professional developers working on companies and wouldn’t be hard to understand that it requires more advanced solutions like Visual Studio because any company developing applications using Unity uses Visual Studio IDE.

The solution I proposed should work, it works for me on test that don’t use unity API and it would work on the other tests if the are executed inside unity itself.

If you don’t evaluate Visual Studio solution (largely used by developers) it’s going to be something that it’s going to be on standby always…

8

@Tomek-Paszek we’ve been able to reach a 20% code coverage on our codebase on .NET API code but we can’t pass that because when I’m going to analyze code coverage involving Unity API it crashes with the f****** “System.Security.SecurityException : ECall methods must be packaged into a system module.” exception.

It there nothing that we can do on our end? A simple workaround/guide/tips/help would be nice. Or it’s impossible without Unity staff involvement?

9

"System.Security.SecurityException : ECall methods must be packaged into a system module." happend because you are trying to invoke Unity API outside of the Unity environment where binding with the navive layer don’t exist (thus “ECall methods…” exception). There is no workaround for that nor a way to simply enable it in VS, for example. Most of Unity API needs to be executed inside the editor or player.

10

That’s why I need that you implement the minimal possible to execute it from editor via command line.

It’s really so much work? Please talk about it internally.

11

Not sure what you expect us to make working? There is no magical button we can switch to turn code coverage on.

12

I know that.

I mean that if it’s possible to ingrate Visual Studio code coverage tool, some other open source tool or you need to develop one on your own.

I only need some technical explanation to understand what’s the work involved in this.

13

To my knowledge, a code coverage tool usually is implemented with profiling the code. That means the tool usually weaves in code marks (during compilation code is added to every method call or even to every line of code). That happens either during compilation or after, where the assembly is modified (which essentially gives you a different, slower version of the same assembly). In case of Unity you also need to remember that it runs on Mono and not every tools that is designed for .Net will work automatically with Mono.

This is not a hard thing do to (from a technical point of view) but it’s messing with a system that makes it very likely to break many other things.

I can refer you to https://feedback.unity3d.com/ which we use to prioritise work on new features.

2 Likes
14

I see.

But seeing that it’s not a hard thing to do it could be a nice challenge for the test runner & qa team to achieve on the upcoming versions.

I opened this thread because in this version I’m seeing movement in this area with Play Mode tests and NUnit library upgrade so I had the feeling that you could listen to my suggestions. If you want to chat about this feel free to do it because I have a Jenkins+Sonar combo on our codebase that is ready to try this when you work on it.

Unity it’s getting bigger and developers are doing more and more complex projects that need tools like this.

1 Like
15

@Tomek-Paszek did you see this?

https://github.com/Microsoft/vstest

Can this help to implement it into Unity faster? Do you have any news? Thanks.

1 Like
16

@Tomek-Paszek I have filled a bug regarding test runner batch mode.

(Case 892798)

Please, check it out.

Do you have any news regarding code coverage? Any internal conversation. Microsoft it’s going open source with many tools regarding code coverage as you can see.