DOTS determinism not yet? Or is it restricted? (some tests)

Hello,
I make simple multiplayer RTS (lockstep) just for my training. Simulation is carried out with a fixed step on each client. Clients send each other only messages about their actions (button is pressed, spell is used, etc.).
But this requires a complete determination of the calculations.
On simple tests (random, division, addition, even LookAt), I saw complete determination and the same results after hundreds of iterations.
Then I started making a simple simulation of enemy movement based on BoidSample and in the process decided to check the determination. And the results did not match at all.

  1. There are 2 spawners on scene. Each spawner creates 1000 enemies in random places. Each spawner has an enemy prefab that has its own ID in SharedComponentData. As a result, yellow and purple capsules are created. All yellow capsules have the same positions as the purple capsules. That is, the input is the same.

Yellow capsules visually overlap purple.


2. All enemies go to the center, trying to avoid collisions (Steer Job). The system performs separate calculations for each group with one ID (as in the BoidSample example).
3. It is expected that after N iterations, enemies with different IDs will be in the same positions. But:

10 seconds

100 seconds
It can be seen that at the beginning all the input data are the same, but after a while the capsules are in completely different places. The more time has passed, the more calculations are done, the greater the difference.
Other systems do not affect enemies.

I know that FloatMode.Deterministic is not ready yet, but should be soon (according to forum posts).
Is there DOTS determinism now? Perhaps with some reasonable restrictions? Maybe I can’t use some math operations? Or do I need to wait while FloatMode.Deterministic will work?
Or am I wrong and my code does not allow determinism?

public class Spawner : MonoBehaviour
{
    public GameObject go;
    public int count;

    public void Update()
    {
        if (!Input.GetKeyDown(KeyCode.Space)) return;
        var settings = GameObjectConversionSettings.FromWorld(World.DefaultGameObjectInjectionWorld, null);
        var prefab = GameObjectConversionUtility.ConvertGameObjectHierarchy(go, settings);
        var entityManager = World.DefaultGameObjectInjectionWorld.EntityManager;
        var rnd = new Unity.Mathematics.Random();
        rnd.InitState();
        for (var x = 0; x < count; x++)
        {
            var instance = entityManager.Instantiate(prefab);
            var position = rnd.NextFloat3Direction() * 30;
            position.y = 0;
            entityManager.SetComponentData(instance, new Translation { Value = position });
        }
    }
}
using System.Collections.Generic;
using Unity.Collections;
using Unity.Entities;
using Unity.Jobs;
using Unity.Burst;
using Unity.Mathematics;
using Unity.Transforms;
using UnityEngine;

[UpdateInGroup(typeof(SimulationSystemGroup))]
[UpdateBefore(typeof(TransformSystemGroup))]
public class BoidSystem : SystemBase
{
    EntityQuery m_BoidQuery;
    List<BoidShipComponent> m_UniqueTypes = new List<BoidShipComponent>(3);

    [BurstCompile(FloatMode = FloatMode.Deterministic, FloatPrecision = FloatPrecision.High)]
    struct MergeCells : IJobNativeMultiHashMapMergedSharedKeyIndices
    {
        public NativeArray<int> cellIndices;
        public NativeArray<float3> cellAlignment;
        public NativeArray<float3> cellSeparation;
        public NativeArray<int> cellCount;
        public void ExecuteFirst(int index)
        {
            cellIndices[index] = index;
        }
        public void ExecuteNext(int cellIndex, int index)
        {
            cellCount[cellIndex] += 1;
            cellAlignment[cellIndex] = cellAlignment[cellIndex] + cellAlignment[index];
            cellSeparation[cellIndex] = cellSeparation[cellIndex] + cellSeparation[index];
            cellIndices[index] = cellIndex;
        }
    }
    [BurstCompile(FloatMode = FloatMode.Deterministic, FloatPrecision = FloatPrecision.High)]
    struct Steer : IJobForEachWithEntity<Rotation, Translation>
    {
        [ReadOnly] public NativeArray<int> cellIndices;
        [ReadOnly] public NativeArray<int> cellCount;
        [ReadOnly] public NativeArray<float3> cellAlignment;
        [ReadOnly] public NativeArray<float3> cellSeparation;
        [ReadOnly] public float deltaTime;

        public void Execute(Entity entity, int entityInQueryIndex,ref Rotation rotation, ref Translation translation)
        {
            var cellIndex = cellIndices[entityInQueryIndex];
            var forward = math.normalize(math.rotate(rotation.Value, new float3(0, 0, 1)));
            var currentPosition = translation.Value;

            var neighborCount = cellCount[cellIndex];
            var alignment = cellAlignment[cellIndex];
            var separation = cellSeparation[cellIndex];

            var alignmentResult = 1
                                  * math.normalizesafe((alignment / neighborCount) - forward);
            var separationResult = 500
                                  * math.normalizesafe((currentPosition * neighborCount) - separation);
            var targetHeading = 50
                                  * math.normalizesafe(new float3(0, 0, 0) - currentPosition);
            var normalHeading = math.normalizesafe(alignmentResult + separationResult + targetHeading);
            var targetForward = normalHeading;
            var nextHeading = math.normalizesafe(forward + deltaTime * (targetForward - forward));
            nextHeading.y = 0;
            rotation.Value = quaternion.LookRotation(nextHeading, math.up());
            var velocity = nextHeading;
            translation.Value += velocity * deltaTime;
        }
    }


    protected override void OnUpdate()
    {
        m_UniqueTypes = new List<BoidShipComponent>();
        EntityManager.GetAllUniqueSharedComponentData(m_UniqueTypes);
        for (int i = 0; i < m_UniqueTypes.Count; i++)
        {
            var settings = m_UniqueTypes[i];
            m_BoidQuery.ResetFilter();
            m_BoidQuery.AddSharedComponentFilter(settings);
            var boidCount = m_BoidQuery.CalculateEntityCount();
            if (boidCount == 0)
            {
                m_BoidQuery.ResetFilter();
                continue;
            }
            var hashMap = new NativeMultiHashMap<int, int>(boidCount, Allocator.TempJob);
            var cellIndices = new NativeArray<int>(boidCount, Allocator.TempJob, NativeArrayOptions.UninitializedMemory);
            var cellCount = new NativeArray<int>(boidCount, Allocator.TempJob, NativeArrayOptions.UninitializedMemory);
            var cellAlignment = new NativeArray<float3>(boidCount, Allocator.TempJob, NativeArrayOptions.UninitializedMemory);
            var cellSeparation = new NativeArray<float3>(boidCount, Allocator.TempJob, NativeArrayOptions.UninitializedMemory);

            var initialCellAlignmentJobHandle = Entities
                .WithName("InitialCellAlignmentJob")
                .WithStoreEntityQueryInField(ref m_BoidQuery)
                .WithSharedComponentFilter(settings)
                .WithBurst(FloatMode.Deterministic, FloatPrecision.High)
            .ForEach((int entityInQueryIndex, in Rotation rotation) =>
            {
                cellAlignment[entityInQueryIndex] = math.normalize(math.rotate(rotation.Value, new float3(0, 0, 1)));
            })
            .ScheduleParallel(Dependency);

            var initialCellSeparationJobHandle = Entities
                .WithName("InitialCellSeparationJob")
                .WithStoreEntityQueryInField(ref m_BoidQuery)
                .WithSharedComponentFilter(settings)
                    .WithBurst(FloatMode.Deterministic, FloatPrecision.High)
                .ForEach((int entityInQueryIndex, in Translation translation) =>
                {
                    cellSeparation[entityInQueryIndex] = translation.Value;
                })
                .ScheduleParallel(Dependency);

            var parallelHashMap = hashMap.AsParallelWriter();
            var hashPositionsJobHandle = Entities
                .WithName("HashPositionsJob")
                .WithStoreEntityQueryInField(ref m_BoidQuery)
                .WithSharedComponentFilter(settings)
                    .WithBurst(FloatMode.Deterministic, FloatPrecision.High)
                .ForEach((int entityInQueryIndex, in Translation translation) =>
                {
                    var hash = (int)math.hash(new int3(math.floor(translation.Value)));
                    parallelHashMap.Add(hash, entityInQueryIndex);
                })
                .ScheduleParallel(Dependency);

            var initialCellCountJob = new MemsetNativeArray<int>
            {
                Source = cellCount,
                Value = 1
            };
            var initialCellCountJobHandle = initialCellCountJob.Schedule(boidCount, 64, Dependency);

            var initialCellBarrierJobHandle = JobHandle.CombineDependencies(initialCellAlignmentJobHandle, initialCellSeparationJobHandle, initialCellCountJobHandle);
            var mergeCellsBarrierJobHandle = JobHandle.CombineDependencies(hashPositionsJobHandle, initialCellBarrierJobHandle, initialCellBarrierJobHandle);

            var mergeCellsJob = new MergeCells
            {
                cellIndices = cellIndices,
                cellAlignment = cellAlignment,
                cellSeparation = cellSeparation,
                cellCount = cellCount,
            };
            var mergeCellsJobHandle = mergeCellsJob.Schedule(hashMap, 64, mergeCellsBarrierJobHandle);
            float deltaTime = math.min(0.05f, UnityEngine.Time.fixedDeltaTime);
            var steerJob = new Steer
            {
                cellAlignment = cellAlignment,
                cellCount = cellCount,
                cellIndices = cellIndices,
                cellSeparation = cellSeparation,
                deltaTime = deltaTime,
            };
            var steerJobHandle = steerJob.Schedule(m_BoidQuery, mergeCellsJobHandle);
            Dependency = steerJobHandle;

            var disposeJobHandle = hashMap.Dispose(Dependency);
            disposeJobHandle = JobHandle.CombineDependencies(disposeJobHandle, cellIndices.Dispose(Dependency));
            disposeJobHandle = JobHandle.CombineDependencies(disposeJobHandle, cellCount.Dispose(Dependency));
            disposeJobHandle = JobHandle.CombineDependencies(disposeJobHandle, cellAlignment.Dispose(Dependency));
            disposeJobHandle = JobHandle.CombineDependencies(disposeJobHandle, cellSeparation.Dispose(Dependency));
            Dependency = disposeJobHandle;
            m_BoidQuery.AddDependency(Dependency);
        }
    }
    protected override void OnCreate()
    {
        m_BoidQuery = GetEntityQuery(new EntityQueryDesc
        {
            All = new[] { ComponentType.ReadOnly<BoidShipComponent>(), ComponentType.ReadWrite<Rotation>(), ComponentType.ReadWrite<Translation>() },
        });
        RequireForUpdate(m_BoidQuery);
    }
}

P.S. The code is bad and was quickly written only for tests.

Your issue lies most likely with float, which isn’t deterministic yet, as far I am aware.
Or is FloatMode.Deterministic confirmed to work now?

You probably should use fixed point approach, or int, where applicable.

Otherwise you loose precision over time.

Edit:

I think here may be also a problem, with order of assigning into hashmap for cells in parallel processes.
Which may break determinism, when try to access it back and trying get random instance from the cell.

I haven’t tested that, but I think may be something to be concerned about.

1 Like

As far as I know, no.

Unfortunately, as far as I know, there are no fixed-point variables and methods in Unity.Mathematics. And my extensions will work worse than float determination (when it comes out).
I was thinking of using int, but I wanted to use NavMesh, and I could not replace the NavMeshQuery API (NavMeshQuery.MoveLocation) with fixed-point precision. And as I understand it, there are no guarantees that MoveLocation will always produce the same result.

I don’t mind using int or other solutions, but I’m worried about built-in functions (like NavMesh) that I cannot change.

I’ll try to wait, I hope it will be released soon.
Thanks.

Edit:

This is very bad, because in one thread this task takes a long time to complete. But as far as I understand BoidSample, the order of the elements does not matter, since all calculations are carried out by cells.

Edit 2:
So, I was wrong. I changed ScheduleParallel to Schedule and removed AsParallelWriter from HashMap. This did not give complete determinism, but the error was noticeably reduced. This still did not solve the problem with float, but showed that the order really affects …

Also, number of available threads, depending on hardware, will most likely affect results.

For that only reason, I may come back to my custom chunks mechanics in my current project, based on native array, even I would like to use hashmap approach.

And as I understand it, the same thing with any parallelism (NativeQueue, CommandBuffer), which will have to be abandoned.

Do you use NavMesh? To prevent units from passing through the environment, I planned to use MoveLocations. But since float3 is used there, I’m afraid of differences on different devices.

Nope. Currently I make different type of games and apply different custom approach, where I don’t need really NavMesh.

I suppose need to wait, until float is fully deterministic.

If purely generating many entities, or removing randomly, than that will be a potential issue.
How I solve that part, unless I can pre-generate entities before hand, i.e. single threaded, I tend to use some form of indexing / reference mechanics, which assigns new entity to their parent, or specific array, keeping relevant entities in order / assignment. So parallelism is not so much of an issue for me, in such case scenario.

As of now, need to implement own system.

I keep my entities positions, scale, and orientation as integers, with scaling up by a factor.

I try to make the entity index absolutely not important in the calculations. As well as the turn of their creation. But now I doubted that this would work.

I already wanted to do it on int, in theory I don’t see any problems with it, but NavMesh … Well, I have to either abandon it and write a more complex system, or wait for support for determinism (second half of 2020? https://discussions.unity.com/t/768750/13 ) It is very sad.

Thanks.

Meaning earliest consideration in 2021. And that with good wind. So knowing how things go, probably later.

So yes, you would need decide, what you require in the project. Weather is worth to build up own mechanics, or wait, but unclear how long.

I started deving with determinism in mind. Haven’t fully tackled the problem yet. But I learned allot in meantime. + number of challenges on the way, as some we discussed here already.

Surely I should run more tests on that matter myself. But I got other priorities atm.

Not sure if I interpreted correctly. However, I don’t use entity index / version as part of determinism. I accept situations, that if one sim will give me entity A and other entity B, they wont affect overall simulation.

That for example was rather simple in my previous spatial grid system, where I populated and moved entities between cells on single thread. But checks could be done multithreaded, always giving me expected results.

Is unclear for me as of yet, if I can trust multihashmap accessing multiple instances per cell, in fixed order, as hashmaps nature does not guarantee that.

I am considering generating and populating cells first, with entities, then generate hashmap, just for fast access by hash if needed. Also, I could only affect part of the grid, and regenerate spatial hashmap, when changes to grid happens, rather on every iteration.

Basically anything in unity is fully deterministic in ECS other than float calculations so using parallel collections and indices and … is all fine howver you need to wait for the burst determinism support for floats to be able to use them.

Dots in general is deterministic. Floating point in burst is deterministic on the same architecture. However… NativeHashMap.Concurrent is not deterministic. The boids sample in general is not deterministic. A simple workaround is to make the job that populates NativeHashMap.Concurrent to be a ScheduleSingle job.

2 Likes

And as I understand it, the same thing with any parallelism (NativeQueue, CommandBuffer), which will have to be >abandoned.
EntityCommandBuffer is also deterministic

2 Likes

When writing deterministic simulation code i suggest you start by making a hashing method of all state you care about especially position and rotation and generate a single hash out of the whole world using bit exact hashing. This is super important to track down which system causes drift. There is all kinds of reasons why you might get indeterminism. If you dont have the right tools setup from the start you are in for a world of pain. All of this is easy to build on top of dots, we have dont some internal prototypes using GGPO ourselves at a hack week, but it’s not something we are currently focused on. For now all those tools you need to build on top of DOTS.

5 Likes

Thanks for your reply. I planned to create a game for PC, without consoles, etc. But as far as I understand, for players with different processors (Intel / AMD, generation?) Float will not give determinism. And this is a huge minus for the multiplayer game.

I’m a little confused (and Google confused me even more): is it true that Intel (x32 / x64) also has different architectures? Or does it mean only the differences between Intel / AMD / x32 / x64? If any Intel processor provides determinism - that would be great!

Burst will generate deterministic code on Intel / AMD. It generates x64 sse4.1 code by default.

3 Likes

That is, all players on any Intel (x32/x64…) can already play with each other with full guaranteed determinism. Like all players on AMD.
Thank you, this is great news!

I already started writing a large post about researching fixed-point libraries, but now this issue is resolved, thanks.

As I understand it, you mean that this is now supported.

Burst does generate deterministic code for x86 between different AMD / INTEL machines.

2 Likes

That is, a player with any x64 architecture processor can play with a player with any x64 architecture processor, like x86 with x86.
And the only limitation: players can have any processors, but with the same architecture.
I’m understood, thank you.

Great information in this thread, nice to hear a bit more in this area and that determinism is still on the roadmap.

It could be good to have a document somewhere listing all the things in DOTs that do not support determinism such as NativeHashmap.Concurrent, NativeQueue etc?

[EDIT] Also maybe the extent of machine architecture coverage?

@Joachim_Ante_1 Is it possible to state in the docs or somewhere in a specific page about which collections and other constructs are not deterministic, as I understand and tried, the NativeHashMap which I wasn’t using is the exception rather t han the rule but still it is good to have it somewhere documented

1 Like