I was wondering if it’s possible to run a specific subset of tests when running them through the command line. I’d like to run most of our tests all the time, but some of them aren’t that critical and I’d rather have them run overnight instead of slowing down our day-to-day work.
We run it all through Jenkins on a build server, so I can’t just select them manually in the test runner in the editor.
The test runner supports a “test filter” option - this is pretty much undocumented (or underdocumented?)
This allows you to run tests that match a certain name using a filter, so it may not be exactly what u need, but you can rename your tests in such a way that will allow you to run a subset.
A more useful feature would’ve been to allow us to specify a category for executing tests via command line. As an old habit, i tag my test fixtures with categories (e.g: integration, slow, fast, etc).
If i could, i would run a specific category (or a set of categories?) when running from the command line.
Another option would’ve been to ignore certain categories (e.g: slow).
But all of these are not available right now from what i know. the test filter is your best bet right now.
Hey. The testFilter is indeed available in order to filter test by names. It is however also possible to filter test by categories using the testCategory argument. Both are things that will be documented correctly in the documentation overhaul we are working on.
The testCategory is exactly what I needed. I think in combination with testFilter that covers all of the use cases I can think of.
Thanks for the answer liortal & Warnecke!
@Warnecke Is there any way to ignore tests or categories?
For example - we have a lot of tests, but some are really slow. i would like our CI builds to run all tests (except those marked with category ‘slow’ or something similar).
On our nightly build we will execute all tests, but for the ones that run many times a day, i would like to avoid running those.
Currently we do not support exclusive filters, but it should be possible for you to filter out some categories by explicitly specify the categories to include. You can give it “Uncategorized” to pick up tests without categories and then specify the categories to include. I believe that you can provide a regex like “^(?!SlowTest).*” (which would match any category that does not start with SlowTest). Note that it is important to pass both “Uncategorized” and the regex as tests without categories does not go trough the regex, but are just matched against the “Uncategorized” string.
@Warnecke Is there any plans to change this behaviour?
NUnit has an include/exclude/combination behaviour in their docs, can’t you mimic that behaviour with test api?
https://docs.nunit.org/2.5.1/consoleCommandLine.html
Beginning with NUnit 2.4.6, you may use a Category Expression with either of these options:
A|B|C
Selects tests having any of the categories A, B or C.
A,B,C
Selects tests having any of the categories A, B or C.
A+B+C
Selects only tests having all three of the categories assigned
A+B|C
Selects tests with both A and B OR with category C.
A+B-C
Selects tests with both A and B but not C.
-A
Selects tests not having category A assigned
A+(B|C)
Selects tests having both category A and either of B or C
It would be an awesome feature imo
Yes. In the up-comming version 1.1.18 of the test framework package we are adding support for using negation on test filters, categories and assembly names.
Bumping this up because we have used category filters for a while, what is essential for CI/CD builds, but we have happened to find a missing feature that we think is so useful and straighforward to implement for test-runner devs: exclude some categories in the project by default.
Say we have categories “Unit, Integration, Component, e2e, Slow”, regardless they are either at PlayMode or EditMode. We want e2e and Slow in some CI/CD pipelines, which is feasible by category filters at command line.
But we want those two categories not only out of some pipelines but out of any local run of tests. At the momend we are forced to manually set the categories using the dropdown — and that’s quite cumbersome.
It sounds to be like you want to mark those slow tests as [Explicit], which will make them only run if the category or another test filter targeting them is given.
Totally! I have missed this NUnit feature for years. I really appreciate your help.