Calculating average data across multiple chunks

Unity Engine
,
1

I’m running into an issue calculating average values across multiple chunks. The data appears correct when the number of entities is contained within a chunk, but once that number of entities spans over multiple chunks, the average calculation becomes inconsistent and changes from frame to frame. I’ve managed to repro it into a single script:

using System.Collections;
using System.Collections.Generic;
using Unity.Collections;
using Unity.Entities;
using Unity.Jobs;
using UnityEngine;

public struct RandomData : IComponentData
{
    public float RandomFloat;
}

[AlwaysUpdateSystem]
public class TestSystem : JobComponentSystem
{
    private NativeArray<float> averagesArray;//0 = total, 1 = count, 2 = average calculated

    private ComponentGroup randomGroup;
   
    protected override void OnCreateManager()
    {
        base.OnCreateManager();
       
        averagesArray = new NativeArray<float>(3, Allocator.Persistent);
        randomGroup = GetComponentGroup(ComponentType.Create<RandomData>());
    }

    protected override JobHandle OnUpdate(JobHandle inputDeps)
    {
        //process results from last frames averages
        Debug.Log($"Total: {averagesArray[0]}, count: {averagesArray[1]}, avg: {averagesArray[2]}");

        averagesArray[0] = 0f;
        averagesArray[1] = 0f;
        averagesArray[2] = 0f;
       
        if (Input.GetKeyUp(KeyCode.Space))
        {
            Debug.Log("Create");
            //create 100 things
            for (int i = 0; i < 500; i++)
            {
                var newEntity = EntityManager.CreateEntity();
                EntityManager.AddComponentData(newEntity, new RandomData()
                {
                    RandomFloat = Random.Range(0, 100f),
                });
            }
        }

        inputDeps = new UpdateGlobalAverages()
        {
            RandomData = GetArchetypeChunkComponentType<RandomData>(),
            AveragesArray = averagesArray,
            FrameCount = Time.frameCount
        }.Schedule(randomGroup);
       
        return inputDeps;
    }
   
    private struct UpdateGlobalAverages : IJobChunk
    {
        [ReadOnly] public ArchetypeChunkComponentType<RandomData> RandomData;

        public NativeArray<float> AveragesArray;


        public int FrameCount;
       
        public void Execute(ArchetypeChunk chunk, int chunkIndex, int firstEntityIndex)
        {
            var dataArray = chunk.GetNativeArray(RandomData);
           
            var dataCount = chunk.Count;

            for (int i = 0; i < dataCount; i++)
            {
                AveragesArray[0] = AveragesArray[0] + dataArray[i].RandomFloat;
            }

            AveragesArray[1] = AveragesArray[1] + dataCount;
            AveragesArray[2] = AveragesArray[0] / AveragesArray[1];
           
            Debug.Log($"{FrameCount}:{dataCount}");
        }
    }
}

Once you hit space more than twice and create over 1000 entities, the average calculation starts changing from frame to frame, despite the data staying static: https://files.facepunch.com/jarryd/2019/04/01/Unity_2019-04-01_21-05-28.jpg

Have I misunderstood chunk processing?

2

IJobChunk is multithreaded and runs for each chunk. Each chunk can store a fixed amount of entities (depend on struct size).
Where you create new entities more than chunk limit then will be created a new chunk so that way you can see two lines from “Debug.Log($”{FrameCount}:{dataCount}“);”.
And now you have two IJobChunk running at the same time and reading and writing the same value without synchronization and because of it, some values can be lost.

Probably replacing NativeArray with two NativeList (one for sum per chunk and another for count per chunk) and then calculating the average in the separate IJob will help.

3

Yeah what ilih said I think is probably the problem.
Also you’re read/write’ing to AveragesArray constantly so very likely to hit contention.

I don’t know if this still valid anymore as I no longer use it but I solved similar problem in past with [NativeSetThreadIndex].
You can try this. This just pseudo, I haven’t tested it.

struct AverageData {
    public int count;
    public float total;
}

onCreateManager() {
    int threadCount = Unity.Jobs.LowLevel.Unsafe.JobsUtility.MaxJobThreadCount;
    averagesArray = new NativeArray<AverageData>(threadCount, Allocator.Persistent);
}

private struct UpdateGlobalAverages : IJobChunk
{
    [NativeSetThreadIndex] private int threadIndex;
     
    public void Execute(ArchetypeChunk chunk, int chunkIndex, int firstEntityIndex)
    {
        ...
        float total = 0;
        for (int i = 0; i < dataCount; i++)    {
            total += dataArray[i].RandomFloat;
        }
        AverageData ad = AveragesArray[threadIndex];
        ad.total += total;
        ad.count += dataCount;
        AveragesArray[threadIndex] = ad;
    }
}

Then you can compute your average from AveragesArray in OnUpdate() and zero out the array again.