Hard dependency on Test Framework 1.5?

Testing & Quality
, , ,
1

Hi,

Recently I bumped the editor version and all my test scripts was damaged. I was using the 2.0.1 version of the test framework but the recent editor versions seem to override it with 1.5. I cannot simply modify the manifest to get 2.0.1

I know that using experimental packages is at my own risk, but why is there such a hard dependency on 1.5? Is 2.0.1 simply abandoned and I should just start migrating my test scripts to 1.5?

Edit: Apparently the 2.0.1 version has been dropped. But the ability to write tests on Assembly-CSharp doesn’t seem to be available in 1.5, which is my major complaint.

2

Yes, it’s no longer in development with any planned changes more slowly migrating into v1.x.

However that should not stop you from adding it to your project. I had 2.x running in my 6.0 project until maybe 2-3 weeks ago. So unless this was a new change, it should still work.

OTOH I have not yet heard that there’s a v1.5 available. Got to check that out.

Is or was that possible in 2.x?
I find that’s bad practice anyway. You gotta be using Assembly Definitions if you have testable code. One major benefit being that usually you only cover some fundamental code with tests, and that kind of code ought to be in an asmdef in order to avoid it being polluted with outside dependencies. This happens more quickly than you can say “autocomplete”. :wink:

3

Yup in Unity 6.0.44 I believe even if I explicitly state that I want Test Framework 2.0.1 in manifest, it still resolve to 1.5.

The thing is that I wrote lots of game logic tests that basically play a short segment of the game (and is using async Setup which isn’t available in 1.5). They are using the [RequirePlayMode] attribute in 2.0.1 and no asmdef was needed. They usually rely on many different systems so splitting them into assemblies is going to be a nightmare and likely will end up with tiny assemblies with only one single script. And that also means any further script might require me to setup asmdef and lots of dependencies.

4

That’s not a unit test, that’s an integration test which should be used sparingly.

Say rather than testing if the player can move forward for a set time, jump on a platform, wait, then fire and confirm it hit the target … what’s that actually testing? And how brittle is that test if any of these parts change, perhaps simply due to design values being changed?

Better unit tests are:

  • move input => wait one frame => did player position change to the right direction?
  • jump input => wait one frame => did player position move upwards?
  • jump input while in air => wait one frame => did player position continue to move downwards?
  • player minimally above ground => wait one frame => is grounded state true?
  • fire input => wait one frame => does the expected projectile exist in the scene, moving in the right direction?
  • setup player on moving platform => wait for end of frame => did player position change the same amount as the platform’s position?

Anything more complex than that makes unit tests brittle and proves nothing because if the test fails, it could fail for an unnecessarily large number of reasons. Integration tests also run far longer than unit tests ie a few seconds vs a few frames. Unit tests should run as fast as possible because you want to run them as often as possible.

5

Right, I suppose I should just bite the bullet and come up with my own framework for those kind of tests. It was just that 2.0.1 suited perfectly for my needs back then and I didn’t anticipated that it would be deprecated and be overriden by another vastly different version, even if the package was not meant for some of my test cases.