public int GetSharedComponentDataIndex<T>(T value)?

[Guedez]

I got a bunch of SharedComponentDatas, and I want to iterate over all entities that match ANY of the datas, so I thought of placing all indexes in a NativeArray and iterating over chunks. But it seems I need an Entity to get it’s SharedComponentData index?
I got a class with internals access to create a method that would do just that, but I don’t quite understand how it all works.

Edit: Any alternative way of getting all entities that have any of my list of SharedComponentDatas is also appreciated.

This is what I have so far but I am getting stumped by a ton of private variables.
so far, full of bugs, don’t work

static int FindSharedComponentIndex<T>(this EntityDataAccess aThis, T newData, ManagedComponentStore mcs) where T : struct {

            var typeIndex = TypeManager.GetTypeIndex<T>();
            var componentType = ComponentType.FromTypeIndex(typeIndex);

            var newSharedComponentDataIndex = mcs.FindNonDefaultSharedComponentIndex(newData);
            aThis.EntityComponentStore->SetSharedComponentDataIndex(entity, componentType, newSharedComponentDataIndex);
            aThis.EntityComponentStore->InvalidateChunkListCacheForChangedArchetypes();
            managedComponentStore.Playback(ref EntityComponentStore->ManagedChangesTracker);
            RemoveSharedComponentReference(newSharedComponentDataIndex);

            var defaultVal = default(T);
            if (TypeManager.Equals(ref defaultVal, ref newData))
                return 0;

            return FindNonDefaultSharedComponentIndex(typeIndex, TypeManager.GetHashCode(ref newData),
                UnsafeUtility.AddressOf(ref newData));
        }

        public unsafe static int FindNonDefaultSharedComponentIndex(this ManagedComponentStore aThis, int typeIndex, int hashCode, void* newData) {
            int itemIndex;
            NativeMultiHashMapIterator<int> iter;

            if (!aThis.m_HashLookup.TryGetFirstValue(hashCode, out itemIndex, out iter))
                return -1;

            var infos = aThis.SharedComponentInfoPtr;
            do {
                var data = aThis.m_SharedComponentData[itemIndex];
                if (data != null && infos[itemIndex].ComponentType == typeIndex) {
                    if (TypeManager.Equals(data, newData, typeIndex))
                        return itemIndex;
                }
            }
            while (aThis.m_HashLookup.TryGetNextValue(out itemIndex, ref iter));

            return -1;
        }

[Guedez]

Yeah, solved with reflection:
code

private static MethodCaller<ManagedComponentStore, int> Delegate;
        private static object[] args = new object[3];
        [NotBurstCompatible]
        public unsafe static int FindSharedComponentIndex<T>(this EntityManager aThis, T newData) where T : struct {

            var access = aThis.GetCheckedEntityDataAccess();
            ManagedComponentStore mcs = access->ManagedComponentStore;
            int typeIndex = TypeManager.GetTypeIndex<T>();

            var defaultVal = default(T);
            if (TypeManager.Equals(ref defaultVal, ref newData))
                return 0;

            if (Delegate == null) {
                Delegate = typeof(ManagedComponentStore).
                    GetMethod("FindNonDefaultSharedComponentIndex", BindingFlags.Instance | BindingFlags.NonPublic,
                    Type.DefaultBinder, new Type[] { typeof(int), typeof(int), typeof(object) }, new ParameterModifier[] { }).
                    DelegateForCall<ManagedComponentStore, int>();
            }
            args[0] = typeIndex;
            args[1] = TypeManager.GetHashCode(ref newData);
            args[2] = newData;
            return Delegate(mcs, args);
        }

Please make ManagedComponentStore.FindNonDefaultSharedComponentIndex either public or internal next release, and while I get that I can just use reflection to get a pointer to ManagedComponentStore.m_HashLookup I really don’t want to have to make sure I have the up to date one in case I remake my world.

[Antypodish]

Using many shared components probably hinders your design choices.
Probably you could combine all shared data’s into single shared data and calculate the hash.
This way you simply would use shared component filter and get all entities of that, without needing reflections, or some weird approaches.
Plus you will eliminate huge fractuating archetypes of entities in chunks.

[Guedez]

I am using Shared Components for spatial queries, so they end up split into small chunks by design. I am hoping for custom chunk sizes to come out in the next 3 years to alleviate the underused chunk issue that comes with it.

[Antypodish]

Have you considered using hash map instead of shared components?

[Guedez]

Not sure how a hashmap would split my chunks spatially… What exactly you propose I use the hashmap for?

[DreamingImLatios]

Why are you not using this? Method GetSharedComponentIndex | Entities | 0.17.0-preview.42

[Guedez]

@DreamingImLatios

Why are you not using this? https://docs.unity3d.com/Packages/com.unity.entities@0.17/api/Unity.Entities.ArchetypeChunk.GetSharedComponentIndex.html#Unity_Entities_ArchetypeChunk_GetSharedComponentIndex__1_Unity_Entities_SharedComponentTypeHandle___0__

While I do use that, that itself does not solve the underlying issue the post was made about.
If I have a bunch of SharedComponentDatas, and I want to know which chunks are set for that, I can’t use the filter because it’s a bunch of them, not a single one.
So I used the hack I created above to convert my list of SharedComponentDatas into a NativeHashMap<int, metadata> which then is used to compare with the shared component index of the chunk I am currently iterating with the SharedComponentTypeHandle
I still need a way to convert a SharedComponentData to it’s related index. How do people even use what you linked without such a function is simply beyond me, what is that even for if you don’t have the indexes to begin with?

Long Explanation

Here is the whole solution I came up with with an explanation of the issue:
I have these sprinklers, they need to find all “waterable” entities within their range of effect, this only needs to be done when they are created or an “waterable” entity is created/destroyed nearby.
So when I tile dirt or remove a tiled dirt block, I fire a GridModifiedEvent, which states which grid it was on. Grids are logical 8x8x8 subdivisions of the world space, my chunks are separated by those so that I don’t need to check literally all entities in the world each time a “waterable” entity is created or destroyed.

In the image, the sprinkers and their respective targets.

The system I devised will, collect all adjacent Tile Grids to the original modified grid, then all adjacents of the adjacents. The smallest group will be for searching entities that might want to know about the new “waterable entity” and the larger group are all the potential targets of all the entities that are searching for entities in range.

The collected TileGrids are SharedComponentDatas, whose indexes are unknowable unless you use the hack I made, are then converted to a list of indexes, which is then used in chunk iteration to collect all relevant entities that will be checked for distance to one another.

Code

using BovineLabs.Event.Containers;
using BovineLabs.Event.Systems;
using System.Collections.Generic;
using System.Linq;
using Unity.Burst;
using Unity.Collections;
using Unity.Entities;
using Unity.Jobs;
using Unity.Jobs.LowLevel.Unsafe;
using Unity.Mathematics;
using Unity.Transforms;
using UnityEngine;
using Unity.Entities.Extension;
using System;

public class GridModifiedEventConsumer : ConsumeEventSystemBase<GridModifiedEvent> {
    public abstract class GridDistanceCollectorGeneric<T> : GridDistanceCollector where T : struct, IBufferElementData {
    }
    private struct EntityPosition {
        public Entity Entity;
        public float3 Position;
        //public float3 WorldPosition;
    }
    NativeHashSet<TileGrid> Grids;
    NativeHashSet<TileGrid> Adjacent;
    NativeHashMap<int, TileGrid> GridIndexes;
    NativeHashMap<int, TileGrid> AdjacentIndexes;
    NativeList<EntityPosition> Collectors;
    NativeList<EntityPosition> Targets;
    NativeMultiHashMap<Entity, Entity> InRange;
    public GridDistanceCollector[] Processors;
    EntityQuery TargetQuery;
    public EntityQuery ExposedGetQuery(params ComponentType[] componentTypes) {
        return GetEntityQuery(componentTypes);
    }
    protected override void Create() {
        base.Create();
        Grids = new NativeHashSet<TileGrid>(64, Allocator.Persistent);
        Adjacent = new NativeHashSet<TileGrid>(512, Allocator.Persistent);
        GridIndexes = new NativeHashMap<int, TileGrid>(64, Allocator.Persistent);
        AdjacentIndexes = new NativeHashMap<int, TileGrid>(512, Allocator.Persistent);
        Collectors = new NativeList<EntityPosition>(4096, Allocator.Persistent);
        Targets = new NativeList<EntityPosition>(16384, Allocator.Persistent);
        InRange = new NativeMultiHashMap<Entity, Entity>(16384, Allocator.Persistent);
        Processors = Utils.GetAllClassesExtendingRecursive<GridDistanceCollector>("Unity", "Microsoft", "System", "Mono", "UMotion").
            Where(T => !T.IsAbstract).Cast(T => (GridDistanceCollector)Activator.CreateInstance(T)).ToArray();
        foreach (GridDistanceCollector Collector in Processors) {
            Collector.CacheQuery = Collector.MakeQuery(this);
        }
        TargetQuery = GetEntityQuery(ComponentType.ReadOnly<TileGrid>(), ComponentType.ReadOnly<LocalToWorld>(), ComponentType.ReadOnly<TileAdjacency>());
    }
    protected override void Destroy() {
        base.Destroy();
        Grids.Dispose();
        Adjacent.Dispose();
        GridIndexes.Dispose();
        AdjacentIndexes.Dispose();
        Collectors.Dispose();
        Targets.Dispose();
        InRange.Dispose();
    }
    protected override void OnEventStream(ref NativeEventStream.Reader reader, int eventCount) {
        if (eventCount == 0 || Processors.Length == 0) {
            return;
        }
        Grids.Clear();
        Adjacent.Clear();
        GridIndexes.Clear();
        AdjacentIndexes.Clear();
        NativeHashSet<TileGrid> _Grids = Grids;
        NativeHashSet<TileGrid> _Adjacent = Adjacent;
        NativeHashMap<int, TileGrid> _GridIndexes = GridIndexes;
        NativeHashMap<int, TileGrid> _AdjacentIndexes = AdjacentIndexes;
        while (eventCount-- > 0) {
            GridModifiedEvent GridModifiedEvent = reader.Read<GridModifiedEvent>();
            _Grids.Add(GridModifiedEvent.TileGrid);
        }
        Job.WithCode(() => {
            foreach (TileGrid Grid in _Grids) {
                Grid.AdjacentForBurst(_Adjacent);
            }
            Utils.Swap(ref _Grids, ref _Adjacent);
            _Adjacent.Clear();
            foreach (TileGrid Grid in _Grids) {
                Grid.AdjacentForBurst(_Adjacent);
                _Adjacent.Add(Grid);
            }
        }).Run();
        Job.WithCode(() => {
            Utils.Swap(ref _Grids, ref _Adjacent);
            foreach (TileGrid Grid in _Grids) {
                int index = EntityManager.FindSharedComponentIndex(Grid);
                if (index != -1) {
                    _GridIndexes.Add(index, Grid);
                }
            }
            foreach (TileGrid Grid in _Adjacent) {
                int index = EntityManager.FindSharedComponentIndex(Grid);
                if (index != -1) {
                    _AdjacentIndexes.Add(index, Grid);
                }
            }
        }).WithoutBurst().Run();
        Collectors.Clear();
        Targets.Clear();

        CollectTargetEntities CollectTargetEntities = new CollectTargetEntities();
        CollectTargetEntities.Filter = _GridIndexes;
        CollectTargetEntities.Output = Targets.AsParallelWriter();
        CollectTargetEntities.TileGrid = GetSharedComponentTypeHandle<TileGrid>();
        CollectTargetEntities.LocalToWorld = GetComponentTypeHandle<LocalToWorld>();
        CollectTargetEntities.EntityTypeHandle = GetEntityTypeHandle();
        JobHandle TargetJob = CollectTargetEntities.ScheduleParallel(TargetQuery, Dependency);

        foreach (GridDistanceCollector Collector in Processors) {
            CollectTargetEntities = new CollectTargetEntities();
            CollectTargetEntities.Filter = _AdjacentIndexes;
            CollectTargetEntities.Output = Collectors.AsParallelWriter();
            CollectTargetEntities.TileGrid = GetSharedComponentTypeHandle<TileGrid>();
            CollectTargetEntities.LocalToWorld = GetComponentTypeHandle<LocalToWorld>();
            CollectTargetEntities.EntityTypeHandle = GetEntityTypeHandle();
            JobHandle CollectorJob = CollectTargetEntities.ScheduleParallel(Collector.CacheQuery, Dependency);
            float distance = Collector.Distance() + 0.1f;
            NativeList<EntityPosition> _Collectors = Collectors;
            NativeList<EntityPosition> _Targets = Targets;
            NativeMultiHashMap<Entity, Entity> _InRange = InRange;
            _InRange.Clear();

            Job.WithCode(() => {//This could be optimized by ordering all collectors and targets by their X and Z positions
                for (int i = 0; i < _Collectors.Length; i++) {//But I can just do that later
                    for (int j = 0; j < _Targets.Length; j++) {
                        if (math.abs(_Collectors[i].Position.x - _Targets[j].Position.x) < distance) {
                            if (math.abs(_Collectors[i].Position.z - _Targets[j].Position.z) < distance) {
                                _InRange.Add(_Collectors[i].Entity, _Targets[j].Entity);
                                //Debug.DrawLine(_Collectors[i].WorldPosition, _Targets[j].WorldPosition, Color.blue, 5);
                                //} else {
                                //    Debug.DrawLine(_Collectors[i].WorldPosition, _Targets[j].WorldPosition, Color.red, 5);
                            }
                            //} else {
                            //    Debug.DrawLine(_Collectors[i].WorldPosition, _Targets[j].WorldPosition, Color.red, 5);
                        }
                    }
                    //DebugExtension.DebugPoint(_Collectors[i].WorldPosition, Color.red, 0.5f, 5);
                }
                //for (int j = 0; j < _Targets.Length; j++) {
                //    DebugExtension.DebugPoint(_Targets[j].WorldPosition, Color.green, 0.5f, 5);
                //}
            }).Schedule(JobHandle.CombineDependencies(TargetJob, CollectorJob)).Complete();
            Collector.ProcessResult(EntityManager, _InRange);
        }

    }
    public class SprinklerDistanceCollector : GridDistanceCollector {
        public override int Distance() {
            return 1;
        }

        public override void DoProcess(EntityManager EntityManager, Entity Key, int Lenght, NativeMultiHashMap<Entity, Entity>.Enumerator Vals) {
            DynamicBuffer<SprinklerTargets> Buffer;
            if (EntityManager.HasComponent<SprinklerTargets>(Key)) {
                Buffer = EntityManager.GetBuffer<SprinklerTargets>(Key);
                Buffer.Clear();
            } else {
                Buffer = EntityManager.AddBuffer<SprinklerTargets>(Key);
            }
            while (Lenght-- > 0) {
                Vals.MoveNext();
                Buffer.Add(Vals.Current);
            }
        }

        public override ComponentType[] GetQuery() {
            return Utils.A(ComponentType.ReadOnly<IsSprinkler>());
        }
    }

    public abstract class GridDistanceCollector {
        public EntityQuery CacheQuery;
        public abstract int Distance();
        public abstract ComponentType[] GetQuery();
        public EntityQuery MakeQuery(GridModifiedEventConsumer source) {
            return source.ExposedGetQuery(GetQuery().Concat(new ComponentType[] { ComponentType.ReadOnly<TileGrid>(),
            ComponentType.ReadOnly<LocalToWorld>() }).ToArray());
        }
        public abstract void DoProcess(EntityManager EntityManager, Entity Key, int Lenght, NativeMultiHashMap<Entity, Entity>.Enumerator Vals);
        public void ProcessResult(EntityManager EntityManager, NativeMultiHashMap<Entity, Entity> inRange) {
            (NativeArray<Entity>, int) keys = inRange.GetUniqueKeyArray(Allocator.Temp);
            int len = keys.Item2;
            for (int i = 0; i < len; i++) {
                Entity Key = keys.Item1[i];
                int Len = inRange.CountValuesForKey(Key);
                DoProcess(EntityManager, Key, Len, inRange.GetValuesForKey(Key));
            }
            keys.Item1.Dispose();
        }
    }

    [BurstCompile]
    struct CollectTargetEntities : IJobChunk {
        [ReadOnly] public NativeHashMap<int, TileGrid> Filter;
        [ReadOnly] public SharedComponentTypeHandle<TileGrid> TileGrid;
        [ReadOnly] public ComponentTypeHandle<LocalToWorld> LocalToWorld;
        [ReadOnly] public EntityTypeHandle EntityTypeHandle;
        public NativeList<EntityPosition>.ParallelWriter Output;

        public void Execute(ArchetypeChunk chunk, int chunkIndex, int firstEntityIndex) {
            if (Filter.TryGetValue(chunk.GetSharedComponentIndex(TileGrid), out TileGrid grid)) {
                NativeArray<Entity> Entities = chunk.GetNativeArray(EntityTypeHandle);
                NativeArray<LocalToWorld> LocalToWorld = chunk.GetNativeArray(this.LocalToWorld);
                for (int i = 0; i < chunk.Count; i++) {
                    Output.AddNoResize(new EntityPosition() {
                        Entity = Entities[i],
                        Position = grid.InGrid(LocalToWorld[i].Position)
                        //,
                        //WorldPosition = LocalToWorld[i].Position
                    });
                }
            }
        }
    }
}

[DreamingImLatios]

@Guedez

I still need a way to convert a SharedComponentData to it’s related index. How do people even use what you linked without such a function is simply beyond me, what is that even for if you don’t have the indexes to begin with?

It is literally in the documentation in the link.

The index value indexes the array returned by GetAllUniqueSharedComponentData(List). If desired, you can create a native array that mirrors your unique value list, but which contains only unmanaged, blittable data and pass that into an IJobChunk job. The unique value list and a specific index is only valid until a structural change occurs.

[Guedez]

Ah, I see, didn’t knew that there was overload that also returned the indexes. While not exactly what I wanted, it certainly would’ve been possible to make do with it, albeit it would be extremely verbose and tedious to compare two lists to extract the related indexes I need from the second one.
This is what I mean

This part would become significantly more complicated and verbose if using such a roundabout way to getting the indexes that have no excuse to not be trivially accessible:

            foreach (TileGrid Grid in _Grids) {
                int index = EntityManager.FindSharedComponentIndex(Grid);
                if (index != -1) {
                    _GridIndexes.Add(index, Grid);
                }
            }
            foreach (TileGrid Grid in _Adjacent) {
                int index = EntityManager.FindSharedComponentIndex(Grid);
                if (index != -1) {
                    _AdjacentIndexes.Add(index, Grid);
                }
            }

I much prefer what I’ve managed to make, even if it uses Reflection, as far as I am aware, the specific way I am calling the reflected method is just twice as slow as if it was a direct call rather than reflection.
Also, calling GetAllUniqueSharedComponentData<T>(List<T>) just for the sake of getting the indexes of about 1(best case) to 36 (worst case) SharedDatas (out of possibly 200+) will get increasingly less and less efficient as the total of SharedDatas increase. I still want a non-private version of the method I am reflecting. And if never delivered, I will eventually use reflection to extract a pointer of that hashmap if this ever becomes a performance bottleneck.

[DreamingImLatios]

I understand the problem now. There are two other options I can think of which avoid reflection, although if this were me I would probably implement a more direct asmref utility.

1. If you know of an entity with the desired shared component, you can use EntityManager.GetSharedComponentDataIndex. It is undocumented, but public.
2. If you don’t know of an entity, you can still get an array of all matching chunks using EntityQuery.SetSharedComponentFilter. Then you can combine all those arrays into one and use an IJobFor to do chunk iteration.

[Guedez]

@DreamingImLatios

I would probably implement a more direct asmref utility.

Do asmref let you access private methods? Were they merely just internal I would’ve already solved it through that.
This greatly interests me.

@DreamingImLatios

1. If you know of an entity with the desired shared component

Unfortunately the whole point of doing this is to get the entities, the fact that this method exists, but the one I want don’t really bothers me.

@DreamingImLatios

2. If you don’t know of an entity, you can still get an array of all matching chunks using EntityQuery.SetSharedComponentFilter. Then you can combine all those arrays into one and use an IJobFor to do chunk iteration.

While that do avoid reflection, would that even be faster? My method is removing all non existing SharedComponentDatas by checking for -1 index, those are usually 70~% of the grids generated by Grid.AdjacentForBurst(_Adjacent);. I have no idea what sort of overhead would having a ton of empty queries allocating empty native arrays is, I highly doubt it would be faster than the chunk iteration I have now.

I guess using reflection means I can’t compile my builds to IL2CPP right? Not that big of a deal since I already don’t use it because any exception becomes a crash as far as I gather through a couple of tests

[DreamingImLatios]

@Guedez

Do asmref let you access private methods? Were they merely just internal I would’ve already solved it through that.
This greatly interests me.

No, but there is an internal codepath that gets you pretty close although not quite optimal. From EntityDataAccess, you can call InsertSharedComponent and that will return you the index. Then after your query finishes, you call RemoveSharedComponentReference for each index. This is effectively what shared component filtering in EntityQueries does.

@Guedez

While that do avoid reflection, would that even be faster?

It would probably be faster for two or three shared component values. But if you get higher then you start to iterate over chunks too many times (even though you only iterate over the entities in a matching chunk once). I had another idea with ArchetypeChunkIterator but I think the first idea in this post would be faster.

@Guedez

My method is removing all non existing SharedComponentDatas by checking for -1 index,

How do you have non-existing shared component data?

@Guedez

I guess using reflection means I can’t compile my builds to IL2CPP right?

I think reflection to call existing functions still works, but don’t quote me on that.

One last idea, wherever in your code you set a shared component value to an entity, immediately afterward you can set a chunk component value to the entity to match the shared component. As long as it is right after, I believe that will always keep the chunk components in sync with the shared components, in which case you can filter on the chunk component values instead. You can then even fast-filter using meta chunk iteration. That might actually be the fastest solution since it keeps everything in Burst.

[Antypodish]

@Guedez

Not sure how a hashmap would split my chunks spatially… What exactly you propose I use the hashmap for?

Boids example use hash map, for spatial mapping.
You can pass key as hash, int. Int2, int3 etc. I.e. Pos xyz and chunk type. That gives plenty options to make spatial mapping, for what you need. Much better approach than fragmenting entities chunks with tons of shared components and their values.

[Guedez]

@Antypodish

Boids example use hash map, for spatial mapping.
You can pass key as hash, int. Int2, int3 etc. I.e. Pos xyz and chunk type. That gives plenty options to make spatial mapping, for what you need. Much better approach than fragmenting entities chunks with tons of shared components and their values.

Ah, you mean to trash the whole “use shared data to spatial partition” and just spatial partition it on a external structure?
Well, this seems to be the best solution after all, the only downside is that I would need to remove deleted entities from this external structure while the shared datas solution don’t need to. I’ve been waiting for someone to drop a easy to use burst enabled octree to do that, but just indexing by 8x8x8 blocks might be fast enough.
Looking at my code, I actually seem to only ever use TileGrid as a shared filter twice, so it might not be that hard. But that’s a problem for future me when I have the time for optimizations on non performance bottlenecks. Them being not on so small chunks might make them render faster too.

@DreamingImLatios

How do you have non-existing shared component data?

This code generates all adjacent TileGrids, regardless of there being any entity there or not. I use it to check if there are actually any entities there.
code

    public void AdjacentForBurst(NativeHashSet<TileGrid> ToAddTo) {
        for (int x = -1; x < 2; x++) {
            for (int y = -1; y < 2; y++) {
                ToAddTo.Add(GetAlignedGrid(new Vector3(Position.x + x * 8, 0, Position.y + y * 8)));
            }
        }
    }
    public TileGrid GetAlignedGrid(Vector3 Position) {
        if (InBounds(Position)) {
            return this;
        }
        TileGrid TileGrid = new TileGrid();
        TileGrid.Position = new int2(ToRange(Position.x), ToRange(Position.z));
        TileGrid.GridOffset = GridOffset;
        TileGrid.Rotation = Rotation;
        return TileGrid;
    }

[Antypodish]

@Guedez

Ah, you mean to trash the whole “use shared data to spatial partition” and just spatial partition it on a external structure?
Well, this seems to be the best solution after all, the only downside is that I would need to remove deleted entities from this external structure while the shared datas solution don’t need to. I’ve been waiting for someone to drop a easy to use burst enabled octree to do that, but just indexing by 8x8x8 blocks might be fast enough.

Pretty much.
NativeHashMap and NativeMultiHashMap both have add and remove methods. I know add is blazing fast.
Don’t know about remove. Depending how often you want to add and remove into spatial map.
Still thousands additoins/removal per frame should be like nothing really while can be both multithreaded and bursted.