Fastest way to get lots of data into entities

[Cell-i-Zenit]

Hi,

i have a big list of data which needs to get distributed to different entities (1 entity gets one entry in the list),

how would you do this?

I got to this point:

1. First create the baseEntity(in my case an object via objectconversion)
2. add all required components to this entity
3. use batch instantiate to create all the entities
4. Use a job to deliver the data (this could be tricky as i need to copy the array into the job (or can i just use a pointer here?)

Any other ideas? I just scaled up my game to the desired object number, but its way to slow instantiating a big amount of entities with the needed data

[DreamingImLatios]

You have the right idea. Make sure you profile steps 3 and 4 individually to find out what might be slow. Also, NativeArrays are actually structs containing pointers to their underlying memory. So assigning an NativeArray to a job field is really just copying the pointer.

[Cell-i-Zenit]

@DreamingImLatios

You have the right idea. Make sure you profile steps 3 and 4 individually to find out what might be slow. Also, NativeArrays are actually structs containing pointers to their underlying memory. So assigning an NativeArray to a job field is really just copying the pointer.

I got the concept a bit more polished:

1. Create the base entity, add sharedComponentData float3
2. Create a subset/all entities via batch instantiate
3. Set the EntityQuery.SCD filter to a specific float3
4. Use an ForEach job to query all entities with specific float3 value (this should be super fast as i have the SCD filter active)
5. Assign all data to each newly created entity
6. reset the scd filter
7. Next frame do the same thing for the next subset of entities with the specific float3
8. repeat this until no more entities needs to be spawned for specific float3, then choose another float3 (there will be ~ 100 different float3’s, each will contain ~ 8k entities

Questions:

1. Does this make sense?
2. Can i reuse the batch insert Array or will i overwrite the entities from the frame before? pseudocode:

onCreate(){
var arr = new NativeArray(1000, type.persistent);
}

onupdate(){
EntityManager.Instantiate(baseEntity, arr);
}

3. Last question: how can i analyse the systems? I tried using the Analyser but all i got was that it told me that system XXXX takes 90% of the cpu resources, but it didnt tell me which method etc

[jonwah00]

Perhaps EntityQuery.CopyFromComponentDataArray will be your friend?

EntityManager.SetComponentData(NativeArray<Entity>,T[] componentData)

What is the most efficient way to set the component data of all entities of a group/in the same chunk/created in the same NativeArray at once? Do i need to iterate over them? Should I use a job instead? I understand that Entities are just ids and do not necessarily lie together in memory but those inside of a chunk do as far as I understand it (isn't that the whole point?). What is the best way to update a whole chunk at once. EntityManager em = ...; int N = ...; for (var i = 0; i < N; i++) …

[Cell-i-Zenit]

@jonwah00

Perhaps EntityQuery.CopyFromComponentDataArray will be your friend?

https://discussions.unity.com/t/789819

I saw that too, but this will make the datalayout pretty shitty… my entities have lots of different components with different data types on them, but the data as i create it currently is stored inside a single class/struct whatever. If i use CopyFromComponentDataArray, i need to slice each class/struct into arrays:

class CoolObject{
        float4: dataForComponent1;
        float3: dataForComponent2;
        int: dataForComponent3;
    }

now i need to have three arrays which contain the data for each component type…

i will think about using that, but iam not sure if i really need that much performance, if it makes my code harder to use

[DreamingImLatios]

@Cell-i-Zenit

I got the concept a bit more polished:

1. Create the base entity, add sharedComponentData float3
2. Create a subset/all entities via batch instantiate
3. Set the EntityQuery.SCD filter to a specific float3
4. Use an ForEach job to query all entities with specific float3 value (this should be super fast as i have the SCD filter active)
5. Assign all data to each newly created entity
6. reset the scd filter
7. Next frame do the same thing for the next subset of entities with the specific float3
8. repeat this until no more entities needs to be spawned for specific float3, then choose another float3 (there will be ~ 100 different float3’s, each will contain ~ 8k entities

Questions:

1. Does this make sense?
2. Can i reuse the batch insert Array or will i overwrite the entities from the frame before? pseudocode:

onCreate(){
var arr = new NativeArray(1000, type.persistent);
}

onupdate(){
EntityManager.Instantiate(baseEntity, arr);
}

3. Last question: how can i analyse the systems? I tried using the Analyser but all i got was that it told me that system XXXX takes 90% of the cpu resources, but it didnt tell me which method etc

1. As long as the new float3 SCD is new and unique, this makes perfect sense and is probably how I would do it too. Otherwise I frequently use a NewGroupTag component instead of the SCD to do the same thing, and then remove the tag component using an EntityQuery when I am done (this invokes a crazy fast path where none of the component data is actually copied to new chunks).
2. All that array contains after calling EntityManager.Instantiate is the list of entities that were created with that particular call. By reusing that array, you are forgetting which entities were instantiated by that call the previous frame. The entities themselves do not get overwritten or destroyed.
3. The Timeline view in the profiler is very powerful at showing you what is going on, especially because it can show you all the worker threads and any job work the main thread waits on.

[jonwah00]

shrug I dunno man, you said when you scaled up to your entity count it was way too slow. This technique requires you to first fill some arrays (a single loop over your list would do it, or you could write extension methods to provide them, or whatever) and then it’s a straight mem copy, I think it’ll be the fastest way you’ll find to do it… anyway up to you

[Cell-i-Zenit]

Iam doing something wrong i guess, since all entities have the same value (0.2630,-0.43908, -8.249)

    private NativeArray<Translation> _data;

    private Entity _baseEntity;
    private int maxNum = 10;

    protected override void OnCreate()
    {
        int i = 0;

        _data = new NativeArray<Translation>(maxNum * maxNum, Allocator.Persistent);
        for (int x = 0; x < maxNum; x++)
        {
            for (int y = 0; y < maxNum; y++)
            {
                _data[i] = new Translation() {Value = new float3(x, 0, y)};
                i++;
            }
        }

        var sett = GameObjectConversionSettings.FromWorld(World.DefaultGameObjectInjectionWorld, new BlobAssetStore());
        _baseEntity = GameObjectConversionUtility.ConvertGameObjectHierarchy(Resources.Load<GameObject>("cube"), sett);

    }

protected override void OnUpdate()
{
    if (Input.GetKeyDown(KeyCode.A))
    {
        Debug.Log("Spawn!");
        
        var arr = new NativeArray<Entity>(maxNum * maxNum, Allocator.Temp);

        EntityManager.Instantiate(_baseEntity, arr);

        EntityQuery query = EntityManager.CreateEntityQuery(ComponentType.ReadWrite<Translation>());
        
        query.CopyFromComponentDataArray(_data);

        Debug.Log("End!");
    }
}

[Cell-i-Zenit]

Ok small update, the code i had was working fine, but digging in the forums revealed that CopyFromComponentDataArray is bugged in version 0.10.0 and doesnt work i downgraded to 0.9.1 and it works now!

The performance is really really good and quite insame … spawning 2k objects per frame is really no problem anymore (i think i will go up much more, but first i need to implement everything.

I have a last question, which iam currently stuck on: What is the fastest way to assign sharedcomponents (in this case rendermeshes), individually on gameobjects?

I spawn 2k objects, but for specific objects i want a different rendermesh.

Currently iam trying to have an array which tells me which object (from the 2k spawned), will have which material and then assign them via a job…

Any better ideas?

[jonwah00]

I’d be interested to know if someone knows a better approach, I’ve been wondering the same thing.

My current approach is just to loop the entities calling SetSharedComponent with a reference to the correct mesh. Which, to be honest, is plenty fast for my use case, as I understand it SCD is just a pointer anyway, so there’s no memory being allocated or copied when you do that.

Just make sure the archetype you’re creating has RenderMesh etc already on it…

[temps12]

@jonwah00

I’d be interested to know if someone knows a better approach, I’ve been wondering the same thing.

My current approach is just to loop the entities calling SetSharedComponent with a reference to the correct mesh. Which, to be honest, is plenty fast for my use case, as I understand it SCD is just a pointer anyway, so there’s no memory being allocated or copied when you do that.

Just make sure the archetype you’re creating has RenderMesh etc already on it…

Setting the shared component data on an entity will make the entity move to a chunk with the new shared component data, so it will make it copy. There might be a way to set the shared component on chunk level which wouldn’t need to make a copy.

[charleshendry]

You can assign it to the chunk by using an entity query
EntityManager.AddSharedComponentData<T>(EntityQuery entityQuery, T componentData)