Today was my first attempt of using the LogAssert class. I had spend a looong time to figure out why my test isn’t working as i expect it to. Here is feedback about this class after trying (and failing) to use it properly today:
First off, i tried looking for “logassert” on the forums and feedback site. Found 0 results… this class doesn’t really look like a popular one at the moment (which is great, since it can be easier to modify it without pissing off too many people!)
All tests have this “hard-wired” behaviour that will basically fail them in case an error is logged to the console. It could’ve been a lot nicer (and cleaner) to be able to toggle this behaviour directly from the [Test] or [UnityTest] attributes. Something like this:
[UnityTest(IgnoreLogErrors = true)]
public IEnumerator MyTest()
{
// ...
}
This behaviour is related to the test itself, why should another class be responsible for turning this behaviour off? also, this way allows for more freedom (turning off for a specific test).
In case i’m ignoring exceptions / error logs (so they don’t fail my test), any Asserts i make will not cause the test to fail as well:
[UnityTest]
public IEnumerator Test()
{
yield return null;
LogAssert.ignoreFailingMessages = true; // with this, asserts below will not fail the test
Assert.That(0 == 2, "0 isn't equal to 2");
Assert.AreEqual(0, 2, "0 isn't equal to 2");
}
This test passes (only tested for PlayMode test, but i suppose it’ll be the same for editor tests as well). Not sure this is the desired behaviour… how can i fail the test now basically ?
The Expect method (!!) is my biggest rant here. since it’s a static method, i somehow assumed that setting it to expect a particular log type + msg will mean that any log messages like that will not fail the test. In reality, it only matches a single log message and then moves on.
For example, in my test i had set something like this:
During my test, any number of messages from that form might be logged, and i don’t want them to fail my test. With the current API, this only matches a single log message, but any subsequent error logs fail the build.
I realize this might be similar to other test frameworks, but it was still confusing to use and i had to spend a lot of time figuring out why it didn’t work.
This can be solved in any number of ways, for example:
LogAssert.ExpectOnce / LogAsset.ExpectAny
LogAssert.Expect(string, int) → expect a log message for the given number of times (-1 for any number?)
Updating the documentation to stress out this fact. Currently this is the description (not sure if it’s clear enough):
Another issue - the Expect method will not fail the test in case a log that matches the expectation is seen, but will fail in case it’s missing. basically, there’s no API to say “don’t fail my test in case this error message is shown”. Should be added somehow to make it easier to ignore certain error messages.
Please consider this feedback when making changes to LogAssert or to the unit test framework in Unity
The test framework and other supporting libraries should probably be open sourced, so i can fix it for myself locally (as well as push those changes over to you, in case you’re interested).
I was surprised to find that that multiple Expect calls will expect the log messages to appear in that order. This ought to be documented and ideally made optional.
Any interesting updates about the test runner or test framework in general? @ElvisAlistar
I am actually using it more often nowadays and i have a few other things that could be useful.
how are you using ignoreFailingMessages? are you setting it up in the beginning of the test?
You’re catching the exception, but it’s still logged to the console?
It’s been almost a year. What’s the delay? Should we continue using UnityTestFramework (since it’s apparently not getting updates) and do our own integrations of NUnit instead?
Core features are still missing - can’t mock MonoBehaviour even using the NUnit features specifcially created for doing so - but when we build this stuff itself (without any of Unity’s code) it Just works. I found a couple more core bugs in Unity 2018’s implementation of the Test framework - stuff that makes it almost completely useless (Unity outputs the wrong error messages and actively corrupts the code its running!). Lost a few hours discovering that. What gives?
i really wish they would open source the whole thing. i am really into unit testing and would love to donate any fixes or improvements to this framework for the sake of all Unity devs!
Internally in Unity we are using the Unity Test Framework heavily. Currently we run 4 million test runs daily on that framework. It is a test framework that we are betting on internally and thus you can be sure that we will keep supporting it.
The iteration time for us to fix issues and apply feedback to the framework is currently not good enough and that is our main motivation to move the framework into a package. It is however, due to our heavy usage of the test framework that it is taking a long time to make the switch to a package. We plan to ship it as a package in 2019.2.
I just want to add a request for having more access to this. I know this isn’t the most active part of the forum - thought I would throw another vote here for package + better access.
The feature that I am really missing is being able to run tests on standalone platform builds without having the editor kick off the tests. Also, it would be great to not need a network connection for capturing the output.
The pipe line I am working in means that I can’t have dedicated editors per each device running the test builds, and I can’t be guaranteed network access between my devices and the network either. In addition not all platforms yet support “Run all in Player” - it would be nice to short cut this.
In regard to the following line causing tests to pass even when asserts fail:
LogAssert.ignoreFailingMessages = true;
I just spent about 4 hours tracking this issue down and came here to post about it. Sad to hear it has been unresolved in more than a year. We have unit tests that cover 3rd party plugins, some of which are logging errors that don’t concern us. Without that line of code, our tests fail due to these errors that we don’t care about. With it, our tests never fail even if they should.
We do use LogAssert but the only use we have for it is Unit Testing our in-game dev console. Therefore we know exactly what LogError input we’re going to pipe through it. It would be helpful to just blanket ignore X number of tests or turn it off altogether. The code in LogAssert.Expect works exactly as (pardon the pun) expected, so porting that to work with LogAssert.ignoreFailingMessages shouldn’t be too hard.
We are working on something for this. It will allow you to split the building of the player from the actual run, thus being able to do the build on a machine without the device and then copy the resulting player to a device/a machine with the device and execute it. We then have some hooks that will allow you to process the test result yourself.
This would be great to send the build to Firebase Test Labs devices. Will the build be compatible with XCTest as well? Since it seems to be a framework Firebase Test Labs supports for iOS build.
A small suggestion, please allow any bundle name other than com.UnityTestRunner.UnityTestRunner. It affects Firebase library in particular and I have to make a new Firebase project with this bundle name and have to manually switch in a custom credential because this lib compares bundle name.
For those following this thread - check out my new Unity Test Extensions project on GitHub.
It contains the LogAssertEx class that provides some improvements that were missing in the original LogAssert.
Right now it offers basic improvements, but will keep on getting more stuff soon
Wow, that’s 3999935 more tests than I run! I think I’m almost catching up
Is there a way to use LogAssert, but have a clear console at the end of the test? I have a function that logs an error when you use it wrong, and a test that uses LogAssert.Expect to make sure the error is logged, but it’s cluttering the debug console every time I run the full test suite. It seems like “pull the error I just checked for out of the log” would be a useful command, at least for my workflow. Maybe it exists and I’m just using it wrong. As liortal said: [quote=“liortal, post:1, topic: 700989, username:liortal”]
this class doesn’t really look like a popular one at the moment.
[/quote] I had the honor of posting the first LogAssert question on the Answers site (as far as my search turned up), but I haven’t had much luck there, so I thought I’d bomb the suggestion thread here as well.
(Just re-reading the thread, I noticed that this question was already brought up. Bump!)
