Batch EntityCommandBuffer

So I read @5argon blog post over here this morning https://gametorrahod.com/unity-ecs-batched-operation-on-entitymanager-3c35e8e5ecf4

And as I use quite a few short lived event entities in my project I was curious if I could write something that could generically replaced EntityCommandBuffer for these. So I did!

Firstly code, this is my first, not well tested and quickly written proof of concept.

-edit- dramatically updating code, will post it when i’m done.
-edit2- latested post here, now 3x faster than original version (10x faster than a barrier): Batch EntityCommandBuffer - Unity Engine - Unity Discussions

How it works. It passes a NativeQueue whenever requested.

Get the barrier with this.batchBarrier = this.World.GetOrCreateManager<BatchBarrierSystem>(); The same barrier is used for all systems.

To create an entity use

NativeQueue<T>.Concurrent createQueue = this.batchBarrier.GetCreateQueue<T>().ToConcurrent();

where T is an IComponentData that will be added and set on the entity.

Then just pass it to a job, for example

        [BurstCompile]
        private struct CreateEntitiesJob : IJobParallelFor
        {
            public NativeQueue<TestComponent>.Concurrent AddComponentQueue;

            /// <inheritdoc />
            public void Execute(int index)
            {
                this.AddComponentQueue.Enqueue(new TestComponent { Index = index });
            }
        }

To destroy an entity, use the

NativeQueue<Entity>.Concurrent = this.batchBarrier.GetDestroyQueue().ToConcurrent(),

Any entity added to this queue will be destroyed.

[BurstCompile]
        private struct DestroyEntitiesJob : IJobProcessComponentDataWithEntity<TestComponent>
        {
            public NativeQueue<Entity>.Concurrent DestroyEntityQueue;

            /// <inheritdoc />
            public void Execute(Entity entity, int index, ref TestComponent data)
            {
                this.DestroyEntityQueue.Enqueue(entity);
            }
        }

The one extra bit you need to do atm is pass the dependency to the system because I don’t have any access to AfterUpdate();

this.batchBarrier.AddDependency(handle);

return handle;

Why bother?

At the moment it is about 3x faster than using EntityCommandBuffers and completely garbage free.
Also it works in Burst jobs

I believe I could probably optimize it quite a bit if I limit the use.

100,000 entities being added and removed each frame.
This is done in a build (with profiler attached obviously)
I’m not sure why endframebarrier is throwing garbage.

21 Likes

Significant performance increase if I force these entities to only live 1 frame (also saves you having to clean them up myself). Has become more like a dedicated event system, which I kind of like.

2 Likes

Moving the SetComponent to a job significantly improves performance (renamed to EventSystem though that kind of clashes with the unity ui system so i’ll probably change that)

Thats 100k entities created in 3.4ms on a pretty old 3570k, vs 34ms for a barrier.

I can only think of 1 more optimization that needs doing now, and that’s making different types set in parallel (won’t affect this benchmark results as they are only a single type test)

2 Likes

Entities will live 1 frame exactly. Destroying entity is now done automatically.

-edit 2018/12/28-

did some unit testing, found some issues and fixed these. also added parallel setting support, should be quite a bit faster.

Source code is now here: ~~https://github.com/tertle/EventBatchSystem~~

-edit 2019/01/09- Read below, repo moved.

3 Likes

@tertle Can you briefly explain how this EventSystem works? I am converting a game from Rx-driven events to ECS and have the most trouble with events.

Sure. So I have a few short lived entities with a single component attached that I use as kind of an event.

For example to request a path in my project, you create a new entity with a FindPath component. This component will be handled by the pathfinding system (though other events might be handled by multiple systems) and then destroyed.

The point of this, I think I called it BatchSystem for the moment as it shared the same name as the unity UI class though it really needs a better name, is to batch create all entity events at once (of the same type, though as I typed this I just realized I could maybe merge the creation of all of event entities at once) instead of creating them individually 1 at a time.

So instead of using EntityCommandBuffer and creating the entity yourself, you BatchSystem.GetEventBatch() where T where the event component. This will return a NativeQueue which every time the method is called so you don’t need to worry about other system dependencies and will be automatically merged. You simply add the event struct to this queue instead and when BatchSystem updates, it’ll create a bunch of entities and set the component data.

All these newly created entities will be stored in the BatchSystem and destroyed 1 frame later in a single batch destroy entity call. You don’t need to worry about maintaining the life of the event entities it’s handled for you. (Whichcan be very useful if your event entities are used by more than 1 system, you don’t have to worry about ordering.)

NativeQueue has a ToConcurrent() method so can be used in parallel but can also be burst compiled unlike entity command buffer.

-edit-

side note, if anyone was using this before today I updated the code with a small bug fix a few hours ago.

5 Likes

Amazing work, thank you

Can we have permission to use your BatchSystem and NativeUnit code?
I’m asking because I’ve noticed the all rights reserved copyright.

Oh yeah of course. They are just auto generated by StyleCop and I’m too lazy to strip it when copy/pasting.

~feel free to use any code I post on unity forums, now or in the future.

-edit-

side note, NativeUnit is really just a NativeArray with a fixed length of 1. More of a convenience.

1 Like

Cool, thanks!

I thought of it as a “NativeReference” :). Even if it’s just like a 1-array, it makes code clearer and I’m all about that!

I just hope someone figures out a way to return a reference, to access direct memory to get rid of the
o = arr[0]; o.var = 123; arr[0] = o idiom for structs.

1 Like

finally got around to doing some unit testing and found some issues (and fixed them).
Also added parallel setting support, should be quite a bit faster. -note- while this is passing all tests, i probably need to do some large scale validation to ensure thread safety.

Source code is now here: GitHub - tertle/com.bovinelabs.entities: A collection of extensions, systems and jobs for Unity ECS.

2 Likes

A little recommendation about repo’s “shape” to support UPM git pulling, if you move the git init location one level deeper inside the Assets folder then put package.json, then anyone can pull that from Unity. It would appears in Project window like other Unity packages. (Wrapped one level in a folder named after the name specified in package.json) dependencies will go in package.json instead of manifest.json. .gitignore will also has to be Assets in this case.

I have 1 Unity project named PackageDev where in the Assets I have multiple folders representing each small package with its own repo. I set a local UPM on each folder so that the change applies to all games, and git would be able to take them online.

3 Likes

Actually I forgot about this feature. I looked at it when it first came out but ran into an issue and totally forgot.

I’ve moved my code to here: https://github.com/tertle/com.bovinelabs.entities if anyone wants.

2 Likes

@tertle could you add a small ReadMe that describes how to get started if it’s possible?

Yeah sure, I’m about to head out for NYE so I’ll do it soon as I get back in the new year.

2 Likes

Happy holidays to you.

1 Like

Added a short readme to GitHub - tertle/com.bovinelabs.entities: A collection of extensions, systems and jobs for Unity ECS.

If there is anything in particular that needs more explaining let me know, i’m terrible at explaining things.

1 Like

Awesome work! Is there a way to add multiple IComponentData to an entity? Also, is it possible to add an ISharedComponentData?

Looks like cool work has been done here.
But got question, since I maybe missed, or misunderstood.
You have mentioned short leaving entities. But is this approach good for creating persistent entities? Sorry, haven’t chance yet, to test the code.

Neither of these are that hard to add but I don’t really see the point of them for this particular sytem.

Yes, if you’re creating a lot of (persistent) entities in a frame you should batch create them, much faster (this system won’t do that for you though.)

1 Like