Why are the tests of the Performance Testing package included in my project?

When I install the Performance Testing package, its own edit and play mode tests are included in my project. Is this on purpose? Should this not be internal tests that you do on your side that are not released as part of the package?

I believe this is fundamentally due to the fact that the package manager doesn't include assemblies with the UNITY_INCLUDE_TESTS define constraint, unless the package type is "tests". And if you specify this category, then all assemblies are included. So the only way to ship assemblies to be used only in tests, is to ship all test assemblies in the package.

I ran into this with my testing utility, and had to use a non-standard layout, by moving the package tests out of the Packages directory to work around this issue. Would be nice if the package manager would support shipping testing utilities without their tests.

Not sure if this is intentional or not (in general, or with the Performance Testing package).

Interesting insight. Would this not be something that could be solved as part of the CI setup of the package project? For example, when building a release of the package, the build script could remove the unneeded test assets.

Actually, it's intentional. One of the important use cases for packages is "I want to modify one of Unity's built-in packages," and so we ship the tests in order to help you make sure that the modifications you make haven't broken anything. They're also, sometimes, a useful reference for 'how do I use this API' - you can look at how the API is exercised in the tests for some insight.

That said, if you aren't modifying a package, we don't want the tests to be in your way - they should not show up in the Test Runner window when the package has not been pulled into your project's Packages folder, for example. If you're seeing otherwise, that's a bug, please report it.

I'm getting tests from the Input system. I know why I get them, but it's kinda annoying.

The input system ships with some testing features (ie. InputTestFixture) which makes it easier to write play mode tests that uses the input system. The problem is that in order to include those features, I have to add this to the manifest:

  "testables": [
    "com.unity.inputsystem"
  ]

Which makes this happen:

6604306--751282--upload_2020-12-9_14-52-4.png

So if I want the testing features, I also need to include the input system's integration tests. Is that the Input system distributing their test features in a silly way, or is it a corner case that's not been considered?

I'm guessing that this is similar to what's happening with OP, can't be sure, though.

Discussing internally, looks like the latter.

1 Like

Putting a package in "testables" indeed has the same effect of specifying the package type as "tests".

But in any case, I'd like to reiterate that it would be nice if the package manager would support shipping testing utilities without their tests. In addition to pure testing packages, I think this is an issue for all framework-like packages, which benefit from having testing utilities (I know at least Zenject has this issue).

In practice, I believe it would require some kind of mechanism of differentiating between test assemblies and test utility assemblies. Currently, I believe UNITY_INCLUDE_TESTS is used to determine the assembly type, but that is not granular enough for this purpose. This data could live either in the assembly definition, or in the package manifest (thinking the latter would be better).

I ran into this issue again, when trying to organise some of our internal packages better. Has the issue been considered more internally, or are we still stuck with either including package tests in consuming projects, or separating test utilities into separate packages?

I'm also running into problems referencing external package test assemblies in my test assembly in packages I'm building.

I wrote some test utilities I wanted to use in all my packages but am unable to do so.

I added testables and asmdef references, but no dice.