Ability to specify custom ChildTests / create hierarchies of tests programmatically

While trying to get an adaptor for our Asset Testing pipeline (not using TestRunner currently) to work, I ran into a couple limitations:

  • there seems to be a hierarchy of ChildTests visible in TestRunner window, but is there a way to create such hierarchies from script and display them there?
  • while [TestCaseSource] with a getter property works, it's evaluated at compile time; I'm trying to test Assets in the Project which means that either

  • I need to recompile the tests every time someone changes anything in the project (not good)

  • I need a way to generate the ChildTests hierarchy before running the tests

  • I need a way to re-evaluate the [TestCaseSource] when running the Tests not at compile time.

Ideally, I'd want to have a single script (test) that acesses the AssetDatabase when clicking "Run Tests" and builds me a hierarchy of little red and green things :)

(we currently have a custom window that does that but I'd like to have an adaptor to TestRunner)

Any ideas?

To illustrate:

4956617--481958--upload_2019-9-12_12-13-22.png

  • this list is only updated on script recompile
  • I'd want "Cube" and "Rattail Fish" to be parent Tests of their respective Asset Tests

Edit: This behaviour (only updating the TestCaseSource when recompiling) actually breaks Unit Testing; running tests after doing some modifications yields broken & wrong inbetween results:
4956617--481961--upload_2019-9-12_12-18-29.png

Note that while some tests fail, some succeed, and some don't run at all, the "overall result" (top pixelated line) is "Passed"!

It does not seem like it is possible with the current version of nunit. The closest thing I could find was https://nunit.org/docs/2.5/suiteBuilders.html , which was never fully realized.

Updating nunit to a newer version is on our radar, but I can not say anything about when it will be done currently.

1 Like

I'm looking forward to this!!

@Warnecke Thanks for verifying this. I think your answer only applies to "hierarchy" part of my question though.

What about making sure the TestCaseSource is queried when actually running the tests compared to at compile time? This actually feels like a bug...

TestCaseSource is evaluated when we build the test tree, which is on two occasions: 1) When we are about to run a set of tests. 2) When we are retrieving the list of tests to display in the ui. If the result of the TestCaseSource differs between those two cases, then the ui will not update correctly. The same happens if you put e.g. values from random into test cases. So far I can see that is the same way as it works if you run the tests in e.g. Visual Studio, Resharper or Rider.

Hm, I can confirm that the above is not true -
TestCaseSource is only evaluated at compile time of the tests currently. Should I file a bug report?

(As for the usecase: I'm testing Assets. This means that somebody can change something in the AssetDatabase, and then I want to press on "Run Tests" and obviously want those changes to be tested against. This is not something that you usually have in Visual Studio or Resharper (a database of things you want to test against).)

I have tried to set up a test that verifies materials based on test case source:

public class MaterialsTests
{

    [Category("Materials")]
    [Test, TestCaseSource(nameof(Cases))]
    public void VerifyMaterial(Material material)
    {
        Debug.Log($"Verifying {material.name}");
    }

    public static IEnumerable Cases()
    {
        return AssetDatabase.FindAssets("t:material", new[] {@"Assets\Temp\Tests\Materials"}).Select(guid =>
            AssetDatabase.LoadAssetAtPath<Material>(AssetDatabase.GUIDToAssetPath(guid)));
    }
}

It seems that the TestCaseSource gets evaluated on every test run, but it is the ui that does not update without a domain reload. I added updating of the ui whenever the test tree is rebuild (whenever any test is run).

In this case it means that after I add a new material and run any test, the new test case will show up.

fineshamelesscoypu

The fix should become available in the next package version.

1 Like

Sounds good. That was a quick fix, thanks for being so responsive!


You are welcome :)

Hey, I see the new package is out, does it contain the fix?

Note: changelog has 2019-11-19 as date ;)
4983779--486089--upload_2019-9-20_14-22-38.png


Yeah. Something went wrong with the date. But this version is a few days old. The fix will be in version 1.1.3

1 Like