Let’s say I’ve created my game, which is a rogue-lite with levels that take 15min to play. To test the rogue-lite element, I would like to simulate a run through the game automatically, by having an AI play the game instead of a human player. Since I don’t need graphical output, I could run this from a shell, and preferably faster than realtime.
I’m having a really hard time getting this to work. Naturally I want deterministic game play, so that is something is wrong with the automatic fast simulation I could plugin the same random seed into the actual level and the watch the AI play the game in realtime to see where things went wrong.
As far as I can tell using the test framework would work here. However:
In Play Mode tests, the game never goes faster than realtime, which means the engine is waiting for something every frame.
In Edit Mode tests not all game systems are updated automatically, so my behaviours aren’t in a state that they can be run.
How can I solve this? I do all movement and such in FixedUpdate() callbacks now, so I expect to run those as fast as possible with my given physics rate (120 per second normally as a test).
This is my PlayMode test:
using System.Collections;
using System.Diagnostics;
using NUnit.Framework;
using UnityEngine;
using UnityEngine.SceneManagement;
using UnityEngine.TestTools;
public class PhysicsFrameCountTest
{
private const int TargetPhysicsFrames = 120 * 15; // Number of physics frames to simulate
private const string SceneName = "TestScene"; // Change to your scene name
[UnityTest]
public IEnumerator GameRunsForFixedPhysicsFrames()
{
UnityEngine.Debug.Log($"Play Mode test started - waiting for {TargetPhysicsFrames} physics frames.");
// Disable rendering bottlenecks
QualitySettings.vSyncCount = 0;
Application.targetFrameRate = -1;
var loadOp = SceneManager.LoadSceneAsync(SceneName, LoadSceneMode.Single);
yield return new WaitUntil(() => loadOp.isDone);
// Wait for at least one physics frame to ensure all components are initialized
yield return new WaitForFixedUpdate();
// Disable automatic physics updates
var prevMode = Physics.simulationMode;
Physics.simulationMode = SimulationMode.Script;
Stopwatch stopwatch = Stopwatch.StartNew();
// Wait until either target frames reached or timeout
while (FixedFrameCounter.count < TargetPhysicsFrames && !FixedFrameCounter.IsLevelComplete())
{
Physics.Simulate(Time.fixedDeltaTime);
var allBehaviours = Object.FindObjectsByType<MonoBehaviour>(FindObjectsSortMode.InstanceID);
foreach (var behaviour in allBehaviours)
{
if (behaviour.enabled && behaviour.gameObject.activeInHierarchy)
{
behaviour.SendMessage("FixedUpdate", SendMessageOptions.DontRequireReceiver);
}
}
}
stopwatch.Stop();
double elapsed = stopwatch.Elapsed.TotalSeconds;
if (FixedFrameCounter.IsLevelComplete())
{
UnityEngine.Debug.Log($"Level is complete real time: {elapsed:0.000} seconds., fixed count: {FixedFrameCounter.count}");
}
// Verify game is still running
Assert.IsTrue(Application.isPlaying, "The game should still be running after the test.");
Physics.simulationMode = prevMode;
UnityEngine.Debug.Log($"Play Mode test fast completed after {FixedFrameCounter.count} physics frames.");
UnityEngine.Debug.Log($"Elapsed real time: {elapsed:0.000} seconds.");
UnityEngine.Debug.Log($"FixedDeltaTime: {Time.fixedDeltaTime} (expected approximately {FixedFrameCounter.count * Time.fixedDeltaTime:0.000} seconds simulated).");
}
}
This code sort of works, but it’s clear there are differences with normal play mode. For example, some of my scripts add new components. Those are found on the next updated and FixedUpdate() is called on them, but Start() was never called.
I’ve played around some more, and it looks like the Time.timeScale option might work for me. The following script seems to call the FixedUpdate() callback a correct number of times:
using System.Collections;
using NUnit.Framework;
using UnityEngine;
using UnityEngine.TestTools;
public class UpdateCounterTests
{
[UnityTest]
public IEnumerator CountUpdateCallsFor2Seconds()
{
// Store original settings
int originalVSyncCount = QualitySettings.vSyncCount;
int originalTargetFrameRate = Application.targetFrameRate;
// Set fixed 60 FPS
QualitySettings.vSyncCount = 0;
Application.targetFrameRate = 0;
Time.fixedDeltaTime = 1f / 60f;
Time.timeScale = 60f;
// Spawn object with UpdateCounter component
GameObject testObject = new GameObject("TestObject");
UpdateCounter counter = testObject.AddComponent<UpdateCounter>();
// Log initial values
Debug.Log($"Settings - vSyncCount: {QualitySettings.vSyncCount}, targetFrameRate: {Application.targetFrameRate}, fixedDeltaTime: {Time.fixedDeltaTime}s");
int startFrameCount = Time.frameCount;
float startWallTime = Time.realtimeSinceStartup;
// Wait for 2 seconds
int countLocal = 0;
float elapsedTime = 0f;
while (elapsedTime < 2f)
{
countLocal++;
elapsedTime += Time.deltaTime;
yield return new WaitForFixedUpdate();
}
int totalFrames = Time.frameCount - startFrameCount;
float wallClockTime = Time.realtimeSinceStartup - startWallTime;
// Log results
Debug.Log($"Results - Update: {counter.UpdateCount}, Frames: {totalFrames}, Time: {Time.time:F3}s, Avg FPS: {totalFrames / elapsedTime:F1}");
Debug.Log($"Results - FixedUpdate: {counter.FixedUpdateCount}, Delta: {Time.fixedDeltaTime:F3}, Time: {Time.fixedTime}s");
Debug.Log($"Wall clock time: {wallClockTime:F3}s, count: {countLocal}");
// Basic assertions
Assert.Greater(counter.UpdateCount, 0, "Update should be called at least once");
Assert.Greater(counter.FixedUpdateCount, 0, "FixedUpdate should be called at least once");
// Restore original settings
QualitySettings.vSyncCount = originalVSyncCount;
Application.targetFrameRate = originalTargetFrameRate;
// Cleanup
Object.Destroy(testObject);
}
}
using UnityEngine;
public class UpdateCounter : MonoBehaviour
{
public int UpdateCount { get; private set; }
public int FixedUpdateCount { get; private set; }
private void Update()
{
UpdateCount++;
}
private void FixedUpdate()
{
FixedUpdateCount++;
}
}
This runs for 0.032s in test, and performs 121 calls to FixedUpdate(). I’ll need to test this in my actual project.
This may work for you, right up to the moment when it slams into a brick wall.
I’m not exaggerating. The moment you do more work collectively in all your FixedUpdate() calls than the computer can do in that interval, the frame update rate goes to SUPER slow.
Do only physics in FixedUpdate, everything else in Update().
I think you need to take this line:
and define what you really want from that.
What are you testing?
What are you measuring?
What are success criteria?
What are failure criteria?
If you’re just running it a bunch to see if it crashes, we generally call this a soak test. I run soak tests all the time. Pure random inputs to the game are often a great start to finding out if the game crashes.
If you’re not measuring and comparing for success or failure, all you are doing is a soak test.
If you want finer-grained tests, you can seed your random number generator to constantly produce the same level and then pre-record a playthrough. Then you can play that back (without any game presentation, just your core game engine running) and verify what happens.
Keep in mind ALL time spent on making crazy tests of your engines is time NOT spent adding new features or fixing your engine. Everything is an economic choice and the only non-renewable economic resource is your time.
Yeah, your game is only as fast as the slowest frame.
Although FixedUpdate and Update will sometimes happen on the same frame and so the total work for that frame is the same regardless. And now that high refresh rate monitors are very common it seems a little wasteful to be doing anything in Update that doesn’t need to be there. InvokeRepeating can still be great for doing occasional updates.
By ensuring your game logic code is independent of Physics and other engine artifacts. You’d run your own game loop which is called from Update() and/or FixedUpdate and gets the Time.deltaTime passed in. Everything that moves things happens internally. After the frame update you synchronize the game state, ie position, rotation, invoke events.
The alternative is to base your gameplay code entirely on Entities (ECS) since that’s deterministic. But it’s also a different conceptual model and workflow and requires getting used to.
That AI would have to be deterministic too. Effectively this means you are always testing two systems at once: the AI providing input, and the game systems performing simulation update. This makes figuring out what went wrong twice as hard.
Instead, you want determinstic playback of a prerecorded game session - which could have been played by AI.
As Kurt has pointed out: what does this even mean? Without a clear definition any attempt at automating tests is pointless. It’s like writing an autopilot software without knowing which plane it will run on.
I don’t see anything that is suitable for such automated playtesting. You can hardly test whether the game is too hard, whether the player progresses too fast, whether the playtime matches expectations, and so forth.
If the concern is primarily about progression then the tool to use is called Spreadsheet. This is where you design and balance your game stats and their progression. Some companies go as far as defining the rules via formulas and export them into their game as part of the game logic.
But there are always factors that make it near impossible to balance a game solely by values. For instance, if pathfinding with obstacle avoidance is used then enemies may be fighting against each other trying to get into a shooting position while others are pushing them back out of their shooting range.
Thanks for the thoughts. I guess the question wasn’t as clear as it could have been.
I was just noodling around and trying to see how fast a run could be done if there is no human behind the wheel. The test framework was just what seemed natural for this, but I’m realizing it isn’t.
The solution I landed on for now is to make a build of my game and run that with the -nographics -batchmode flags. I’ve added an argument that can set Time.timeScale from the commandline. The ‘game’ runs for the equivalent of 15 minutes then quits automatically, which on my PC takes about 15s at Time.timeScale = 500. Logging some parameters to the commandline lets me check that it matches what happens when I run the game normally.
As mentioned, I need to ensure all randomness comes through a single generator, or at least that the generators are seeded properly each time. And yes, this process does indeed include everything: AI, physics, logic, which is what I want in my case since I’m not so much testing as Monte Carlo simulating the game.
I’m not good at designing numbers, but I was wondering if I could get the simulation this fast so I could run it many times with small permutations of numbers I think are good. If the stats at the end don’t match what I want (since the goal is a runtime length, or an enemy count killed or some such) I can adjust.
My game loop isn’t doing much yet, so I do wonder how this will hold up as I implement a more realistic game. (My inspiration for this entire project was Ball x Pit)
One thing that works great for any game using play mode tests is setting Time.captureFramerate to 30 (or 60) during test start.
This makes the game think it’s running at a fixed 30hz, all your timesteps look normal, you don’t have to do any test-specific work, but it decouples the game from the framerate. Frames will happen as fast as they can, and rendering will happen whenever possible. In effect, on a good PC, you’ll see the game zoom.
Conversely, on a poor PC, your tests won’t fail because of inconsistent framerates. You’ll get deterministic tests across any machine.
You can set this up once for all your tests using [SetUpFixture] SetUpFixture | NUnit Docs and calling Time.captureFramerate=30 in its [OneTimeSetup].