Deterministic Physics for Lockstep Networking, Any Progress?

Unity Engine
1

Hi Unity Community,

I’m working on an open source lockstep networking layer (atop socket.io) built for large scale RTS style games requiring thousands of units moving dynamically in real time. We already have all the server and lockstep code working perfectly fine, the only issue is the non-deterministic physics engine that causes floating point calculation entropy to degrade the game state too quickly in order to maintain synchronicity across multiple clients.

I’m writing today in hopes of finding others working on this issue whether that be by virtualizing floating point calculations deterministically like Java with a special compiler flag (slower at crunching, but faster at syncing), or by running an integer based physics engine instead to maintain determinism. I’ve also read of one person attempting a type of “floating point correction” with no luck.

Until we can achieve deterministic simulation we are limited to roughly 100 networked game objects in real time - consider that the original Starcraft could handle upwards of 1200 networked units in real time no problem (and that was over 15 years ago). Determinism is a powerful feature to add for game developers, and until it exists in Unity we can’t really make a great RTS with Unity, without altering our design radically to the point where it wouldn’t be the same game.

If anyone at Unity can give any details towards determinism in the engine it would be greatly appreciated!

Thanks,

Adrian

1 Like
2

Don’t get too discouraged - determinism was a big problem early on in StarCraft, as replays would often fail due to very similar problems. This is a hard problem, so good luck to you. My five cents:

The Unity engine has floating-point coordinates at the core of everything, and uses a non-deterministic physics engine. This means that trying to position networked transforms based on delta positions is bound to fail. AFAIK, all the big multiplayer games out there sends absolute positions from the server, and uses interpolation code on the client side to make stuff look smooth.

To put it like this - even if you had a deterministic simulation running both on the server and on the client, dropped and corrupted packages would eventually lead to degradation of synchronous states anyways. You’ll have to correct for errors, so writing solid error correction and interpolation code for the client side will probably be a lot better of a direction to go in than to try to find a deterministic solution.

If you really want that determinism, you’re going to have to find a stand-alone framework that allows you to create a deterministic model, and then use your game engine as a view of that model, instead of using the engine for everything.

3

Thanks for the reply Baste, I’ve certainly read a lot about starcraft (and age of empires) moving through their growing pains. AoE had bugs right up until release caused by a horse not synchronizing its starting angle and slowly tearing the game state apart.

Java based engines can use floats for everything too, but Java has a compiler flag to do software based deterministic floating point calculations (which should be the same no matter the hardware). Unfortunately this isn’t the case for .net:

I’m also aware that most every online game today (less Starcraft 2) uses a continuous-synchronizing system, with a layer of branch prediction and another layer of interpolation for smoothing.

Regardless of the issues other RTSes suffered, they made it through in the end and released games with massive unit counts and minimal network overhead. As far as current networking technology goes, you hit a hard bottleneck around 50 synchronized units for anyone without a fiber optic connection (aka 99% of the world), and another bottleneck around 100 units for CPU based parsing/packing over the network.

It looks like we will be shimming in an RTS specific integer based physics model to remove floats from the logic part of the equation. That will solve the non-deterministic issue for our use case, as the view will receive its float positions from the integer model.

The next issue is dropped packets, corrupt packets, latency spikes, etc. But I would argue that this problem is remedied in the same way in starcraft and gta5 online - if a player falls out of sync they pretty much need a fresh authoritative copy of the whole game state - and I would certainly rather send a whole state once in a while, rather than a continuous torrent of state updates.

Well, thanks for your help again Baste! There are so few people “down here” working on this problem, or even aware of this problem that it can be difficult to get feedback at all!

We just released v0.1.0 of “Lockstep.IO” (built on Socket.IO) which currently handles the lockstep issue quite well (but not yet determinism). We chose Socket.IO for its consistency in packet delivery, and advanced “long polling” logic that comes built in:
https://www.youtube.com/watch?v=HN_oLZy5tYc

We did some consistency tests and things break down really quickly:

Switching to an integer based model should correct that remaining consistency problem giving us true “deterministic lockstep” and allowing us to push our RTS closer to 1000-2000 units in real time over a network.

Again thanks for your help, we are making all our of networking code open source in hopes of bringing on a new wave of multiplayer RTS games in Unity! Hopefully!

4

deterministic synchronized lockstep impossible in unity because of float point physics
So…why hasn’t unity fixed this after so many years? They would be the only engine that could do RTS games ?_?

5

To add more have you guys added a sync check every now and then as a workaround?