Hello everyone! As I promised @Joachim_Ante_1 
I wrote a little about our experience of moving to ECS and the Job System in a combat project, refactoring, the difficulties we faced and of course the pleasant results we achieved.
(I apologize in advance for my English 
)
(And also you can read this post on Medium or Unity Connect)
And so, for many it’s not a secret that Unity set a new course for the development of the engine (in my opinion very true, effective and really cool), innovations of the engine family of the 2018 engine brought a huge heap of features, additions and really important changes, one of the most “powerful”, and I think everyone will agree with me, these are new render pipelines representing new, extensive opportunities for graphic programmers, or rather slightly simplifying their lives, by extending the rendering capabilities from managed code and going under the auspices of SRP — Scriptable Render Pipeline, and the second global innovation management, about the use of which in production I am today and tell you — ECS and Job System.
I think in the form of a small introduction, it is worth to tell a little about what this design pattern is, very popular in the gamedevelopment, and also about our project, which we successfully put on the rails of ECS and Job-s and continue to develop new ideas for this approach, because almost every week Unity developers bring fresh meat… in the sense of fresh changes, additions and opportunities (often these changes were rolled out very timely, making it easier for us and letting us not write our implementation of certain features, be it Culling, Hierarchy, and other).
So, what is ECS? Entity-Component-System is an architectural pattern widely used in the gaming industry, as you can guess from its name, it consists of three key pillars: Entity, Component and System.
I’ll just leave it here…
Entity — is nothing more than an identifier, which determines and indicates the existence of some object, the entity itself, with which systems can be operated, which can be assigned certain properties by means of components. The component is a data container, without any logic, a certain set of parameters related together and defining properties that you can, let’s say, give to our entity. One of the examples taken from our game can be — resources, groups of entities with Resource component, which has resource quantity fields, its type, the entity with which this resource is associated, etc).
Component group for injection
The system is the part that is responsible for the operations on the data, in other words if the entity is an identifier, the component is data, then the system is the action to be performed on certain entities with certain properties (components), all these 3 pillars are independent of each other, which gives our code incredible flexibility, modularity and scalability, we take another set of entities, and the system will also operate with them, set other sets of components — again everything continues to work, this is one of those big pluses e gives us the ECS — great modularity and reusability of our code. Now, presenting in general terms what ECS is, you can ask yourself — “But, I can implement this pattern in my project on OOP, I can define component classes, I can abstract logic aside from all this, I can create my definition of entities and work with this!”, and to some extent this is true, and here comes on stage — Data Oriented Design (DOD) and Job System. An important feature of the new approach implementing ECS in Unity is that its implementation is built around the DOD principle, a good and simple explanation can be found of Mike Acton talk from GDC 2018:
The essence of this approach, in simple terms and as I understand it, is to organize the data in such a way that they are cache friendly, ie went sequentially and to the maximum were placed in the processor’s cache line, avoid loading useless data (ie having a structure with 10 fields and using only 1 field, it would be better to take this field into a separate structure and load it), which reduces the memory accesses and thus gives us a performance boost.
The second “kick for speed” is the Job System (Unity has already transferred the manual to the official documentation from github, you can find it Here, as well as basic acquaintance with ECS you can already find on the official site in the training section) and developed for it — Burst Compiler, which compiles the C # / IL code into high-performance native code, and as well as the new Unity.Mathematics library, also optimized for native code, SIMD and Burst, this system allows you to write multithreaded code is fast and easy (in comparison with the classic Thread approach of course), under the hood the system has a good security implementation that will save a lot of race conditions, all work is done on the Worker Threads, the Job System for us operates with available resources and distributes Job-s on them, organizes Lock-and etc. Summing up all of the above, we can distinguish for ourselves the key advantages:
- Huge performance
- Easy multithreading
- Modularity and scalability
- Available from the box
All this can be perfectly described by the tagline which, lately, very often sounds from representatives of Unity - Performance by default.
And so getting an idea about the new systems, let’s talk about applying them in practice. Despite the fact that Unity warns that use in production — at your own fear and risk, we could not bypass such significant and very steep changes that were very suitable for our project, and therefore began to refactor everything from the very first previews of versions ECS and the Job System. As I said above, these innovations are very well placed on our game — Elinor.
Our game (early alpha gameplay)
This is a single player real-time strategy in a realistic setting with a small amount of fantasy. The player has to develop his economy, train the army and build strong walls to defend himself against a raid of innumerable amounts (more than a few tens of thousands at a time) at night, and therefore the defending troops must be proportionate to the attacking horde, which leads us to the total army volumes up to hundreds of thousands of units at a time. The economy is not a key aspect, but it is extremely important at the initial stages of the city development in order to establish a continuous production of the army, it is implemented in the form of indirect management of resource flows with full visualization of the entire process of production of game values (and this, for a second, tens of thousands of objects in the game world with its own set of data). Hence again, we highlight the key requirements, and compare them with the advantages that the ECS and Job System give, and which have been described above:
- Dozens, even hundreds of thousands of units
- Tens of thousands of visualized resources, population, etc
- 60+ FPS, by itself
And as we can see, the bonuses from using the new approach are perfectly superimposed on our requirements, and therefore the use of ECS and the Job System is obvious (well, nowhere without my craving for learning new features of course).
Perhaps the most difficult thing in applying a new approach is to reorient your brain from the classic OOD approach to DOD, working for many years as a .NET developer on projects built entirely on OO design, it was not easy enough for me, at that time, a sufficient amount of documentation, the forum was just beginning to come alive, and therefore many things had to spend a lot of time learning examples from the repository, reviewing the performances with GDC, Unite Berlin, and actively communicating on the forum. Therefore, “moving” started with a simple, try out the Job System, which, in principle, is quite autonomous from ECS and can be used in the classical approach.
The first thing that was decided to carry out on parallel calculations is the generation of a mask for the grid construction shader.
Initially it was necessary to receive an array of pixels in the old manner, then copy it to NativeArray and send it to IJobParallelFor to process the pixels in parallel, then reverse-order the NativeArray into an array and assign it to the texture, this approach gave a slight increase, but not as expected, since the conversion of the array back and forth covered almost the entire bonus of parallel computing. But after a while Texture2D had 2 wonderful methods GetRawTextureData / LoadRawTextureData working immediately with NativeArray and rid us of the dances with a tambourine with the usual array, from this moment the process of mask generation became almost invisible for us, according to the resources spent.
But this is only the beginning, so it’s time to use all the innovations in a complex way and then we were expecting another underwater rock — in view of the fact that ECS, at that time, was in a very early preview (although now in the preview, but the functionality has already grown significantly) and it simply did not exist many basic things we used, be it the Colliders, NavMesh components, standard renderers, and so on. At first this is very unsettling, and therefore not all things could be immediately transferred to pure ECS. For this reason, many systems implemented in the game had to be translated into a hybrid approach. In a hybrid ECS, we do not get a significant performance boost (only small, on average 5–10% but it’s better than nothing), but we have the ability to work with classic components in systems (I’ll make a reservation, under the hybrid approach I mean using ComponentArray and GameObjectEntity, the rest of the implementation I use the same as with the pure approach, but the hybrid approach has limitations, we still can not use the classical components outside the main thread, so the path to Job-s is closed to them). On this approach, the system of citizens is implemented in our game. Every citizen is a regular GO with a set of classic components — Rigidbody, Collider, NavMeshAgent, etc. But it’s also worth mentioning separately the component Citizen,
in this component we store different state of a citizen, determine whether he is a builder or a peddler or another type of employee, what resources he carries and whether he carries in general, etc. and we can use such a structure in the Jobs, which we actually did, thanks to such a “dirty” hack, our calculations of the number of citizens in the city, the transitions of their states, birth and death, verification of targets for transfer / collection of resources, etc., occur very quickly (Don’t forget that this is only for a hybrid approach
)
But this is only a temporary solution, now we are actively completing the core for navigation and physics on the basis of Spatial Hash Map and jobified RaycastCommand and NavMeshQuery (based on an excellent example from Nordeus) after which the civilian population will be transferred from a partially hybrid approach to clean ECS.
The most interesting part begins with writing systems on pure ECS. At this stage, we have such systems: Displaying the surrounding forest (as we remember from the description, all the resources we have are real objects, which means the trees too, because they can be cut for processing, each tree is an Entity with a Wood component,
Wood system
Just for understanding world size
displaying all resources in the game, displaying walls and roads (or rather the mode of their construction, in view of the fact that there is a lot of under the hood calculations, to correctly display the orientation of the walls, their rotation etc.). The pure approach has a large number of requirements and rigid limits in the writing of code, which, again, is due to the tagline — Performance by default. But due to these limitations, performance really becomes transcendental. Thanks to Burst compilation, iterating over all the resources in the game, displaying them, moving them, adding and destroying them becomes quite simple and very fast operation, even though we can have tens of thousands of different resources on the screen at once. its parameters.
As a result, after implementing a new approach, we can work with a much larger number of objects than before, performance has grown many times, it has become possible to quickly and easily parallelize multiple flows without worrying about threads, gameplay features that you think you can implement with less effort.
In my opinion, the fact that ECS is now in active development does not in any way prevent us from starting to use it in production, for our example it is clear that we can now practically implement all the necessary functionality in the game, we did not notice serious blocking problems throughout the transition , and if they did, either in the next update of the package it was fixed and many new useful things were added, or you could find a temporary patch on the forum (or just come up with your own workaround). A huge plus that gave us early access to the ECS and the Job System is precisely the understanding of how these systems are arranged inside, how they evolved from the very beginning, why some systems are implemented this way, and not otherwise, why some things were rewritten or abolished, this allows you to better understand the new system and easier to navigate in it at a low level.
If you compare the state of ECS when it was just born and now, the differences are huge. The new versions include the system of Culling, prefabs, Dynamic Buffers, Chunk Iteration, rewrited Transform System, Reactive System, Concurent EntityCommandBuffer, LODs and other amenities that make life easier for us, and without which it was difficult at first, and therefore to join a new approach and starting to study it now is much easier, and in view of the fact that behind this approach the future of the engine and its vector of development, everyone should already pay attention to the ECS and the Job System, and start gradually to delve into them.
If I’m wrong in some points, let me know 
