Hello,
I want to try automating some of the multiplayer testing. I have a series PlayMode tests that load scenes, run scenarios and report any errors. I wanted to try converting some of them to work in multiplayer.
I’ve been playing around with Multiplayer Play Mode and I was able to have a Play Mode test run on both my main editor and an additional Editor Instance. Any error (from either Main or Client) would cause the test to fail which is exactly what I want.
At this point I’m struggling with the automation aspect and I feel like I hit a wall. I’ve been experimenting with having both Scenario Contol and Manual Control for my Client instance and using reflection to launch it. However I think that reallistically only Manual Control is viable, as Scenario Control would require launching the additional instance before entering Play Mode. Since I want to run a mix of SP and MP PlayMode tests I don’t want the client instance running for SP tests.
For manual control, I essenially tried “clicking” the activate button in the status window from code. While this launches the instance, I get tons of error related to communication which interferes with the test. I saw that 6.4 added support for controlling how Play Mode is entered, but I’m not sure if that’s gonna help with launching/closing additional client instances.
If anyone has sucessfully used MPPM to automate some of their Play Mode tests I would love to hear how it’s done.
I assume you refer to this, so for reference respectively in case you didn’t mean that post:
In essence, it’s best not to waste any time on trying to get automated multiplayer tests to work with MPPM. Or connected processes in general.
We call these integration tests, not unit tests, as they require heavy lifting before the test even begins (eg setup networking, connect clients, launch a scene, wait for everyone to have connected, ..) and they are brittle and require a high degree of maintenance due to the many interdependent systems being involved in that test. You usually cover most of those tests during everyday dev/play cycles anyway: “Connecting still works. Guns still fire. Projectiles still deal damage. Etc”.
It’s best to mock such tests ie assume there is a connection and some clients. However this requires engineering the code with mocks in mind, which means some form of abstraction involving at least interfaces and deferring netcode logic execution strictly to custom classes used by NetworkBehaviour, whose [RPC] methods merely become dispatchers:
Then you can run such tests much faster and just assume “this RPC went from client to server and back” and see that the player received damage from another client’s actions.
This may seem like a lot of extra work, while not actually testing the real thing, but if you want to lean strongly into testing this solution will reap benefits in the long term. You’ll find the netcode also becomes much cleaner and less brittle with the strict separation and test-first approach. What you ultimately skip is the things that should work anyway: connections and sending messages across the wire. You can also simulate latency just by queueing a “mock RPC” for a couple frames before calling the target method.
As mentioned above, MPPM is not really in a good shape right now to run automated tests with. It’s our goal to make it useful for such use cases, and you can start seeing efforts in that direction in Unity 6.4, but as you’ve experienced it’s not exactly ready for that yet. I’ll bring it back up with the team however. You’re like the third person asking about this in a month which would tend to indicate we might have under-estimated the demand for it. Maybe we can reshuffle some priorities to have something useful for automated tests sooner.
In the meantime I would suggest following the advice from @CodeSmile above. And if you’re using Netcode for Entities, you could also look at testing with thin clients or running multiple worlds in the same process which should alleviate the need for multiple editor instances like in MPPM.
I am curious about these communication errors however, because “clicking” the button from a test is how we test the UI interactions in our own tests and we haven’t observed that. Could be there’s a bug that needs fixing here…
I spent a bit more time working on this and I can more or less start a secondary instance automatically through code. I still get some errors as shown in the screenshots below.
As for my setup, I have a 3 scene setup:
Startup Scene used for bootstrapping
Logic scene where all the game systems live
Environment scene with all the 3D assets
I currently have a custom attribute for Play Mode tests that does the following:
Modify some of my game settings, to enable awaiting for clients to connect
Modify the MPPM config to switch from Default to my custom one
Activate the secondary instance trought reflection
I will probably deprecate this attribute in favor of OneTimeSetup as it makes more sense.
I have another attribute that loads my my first 2 scenes via EditorSceneManager.LoadSceneInPlayMode. Since this is done before the test starts, I cannot use NetworkManager.Singleton.SceneManager.LoadScene, however it seems that it works for now.
Once the second instance launches, the scenes get loaded there as well. The host instance proceeds to load the environment and executes the test. Since this happens at runtime, NGO handles loading and syncing the ENV scene with the client(s). When the ENV scenes is loading on the client isntance I get the following errors there, but everything still loads.
@CodeSmile thank you for the suggestion. My project isn’t a typical use case for NGO. All the work is done by the host instance, while the clients are simply used to render stuff. It’s mostly Network Transforms and I don’t really have Rpc methods. I’m more interested in testing the spawn and initialization logic rather than “gameplay”, as “gameplay” only runs on the host and is already covered by a mix of unit/integration tests.
That GetBool exception is a bit unexpected because that code should never run outside the main thread. In any case that might be something we can do something about since I’m not sure that call is truly needed here. Will look into this. For the TLS error, that’s way more lower-level and I’m not sure I can offer any sensible advice without digging deeper.
I discussed this with a colleague and we’re thinking that perhaps doing all this from an attribute could be causing issues here. Like maybe the attribute is executing in a context where not everything is ready to set up MPPM instances and switch scenes. If you do test with OneTimeSetup instead, please let us know if it solves your problem.
FYI we just posted about our plans to improve the integration of MPPM with our test framework. We’d be grateful for any suggestions or feedback you could provide, since that’s something you’ve been playing with.
