DOTS development status and next milestones - November 2023

Unity Engine
, ,
1

Hello, everyone — we’re here with our Q4 2023 DOTS roadmap update!

What is DOTS?
Our goal with Unity’s Data-Oriented Tech Stack (DOTS) is to empower creators to build more ambitious games with Unity and serve their needs in the production of complex multiplayer, crossplay, and open-world games.

DOTS consists of several packages based on the Entity Component System (ECS) architecture. The notion of DOTS is often extended to include the Burst compiler and C# Job System, two technologies that have been created to support the development of ECS but are independent of it. The Burst compiler and the C# Job System can be used in any Unity project, regardless of whether ECS is being used.

This past June, we celebrated the Entities package officially entering Released status with Entities 1.0, ensuring support for production on projects using the 2022 LTS. More recently, we released Entities 1.1, compatible with the 2022.3 LTS.

If you’re looking to get started with DOTS (or just need a refresher), we recommend watching our DOTS overview from GameDevGuild 2022 and checking out the following list of resources and studio production stories:

  • Entities API Overview, ECS Samples and Learn Content: The Entities API Overview, ECS samples and Learn Content reflect the latest APIs available with ECS for Unity in a bite-sized format. These resources will be especially useful for new users to learn about fundamental concepts of data-oriented design.

  • ECS Network Racing: If you’re looking for a smaller-scale multiplayer sample with ECS for Unity, ECS Network Racing showcases the capabilities of Netcode for Entities and how you can easily build a multiplayer game out of the box. Specifically, the sample exemplifies an implementation of client/server architecture with client-side prediction, interpolation, and lag compensation.

  • Megacity Multiplayer: Megacity Multiplayer showcases how creators can build ambitious multiplayer titles that support more than 64 concurrent players, while leveraging UGS services like Game Server Hosting, Authentication, Voice Chat, and Matchmaker.

  • Experimental Character Controller Package: For users looking to quickly jump into building their game, the experimental character controller package also contains bite-sized samples illustrating how you can quickly drag and drop the experimental character controller into your scene, and get prototyping quickly.

  • Stickylock Studios - Histera (March 2023): Jamel Ziaty, producer from StickyLock shares their journey and experience using ECS for Unity, Netcode for Entities, Game Server Hosting, and many more Unity Gaming Services for the creation of Histera, a competitive FPS that mixes prehistoric settings with war-torn 20th-century cities through time travel.

  • Sunblink Entertainment - HEROish (March 2023): Justin Larrabee, CTO at Sunblink Entertainment, shares how their clever use of ECS for the production of HEROish helped them achieve performance, reduce production risks, and empower designers.

  • Walaber Entertainment - JellyCar Worlds (December 2022): Tim “Walaber” Fitzrandolph, Solo Developer at Walaber Entertainment dives into using ECS for Unity to create a soft-body physics system for the JellyCar title.

  • Kasedo Games - IXION (August 2022): Technical Director Jérémy Guéry walks us through how ECS has powered heavy NPC simulation for their city builder in space.

  • Stunlock Studios - V Rising (June 2022): In a 3h-long Twitch stream, lead developers Rasmus Höök and Fredrik Haraldsson presented their use of ECS, from world-building in the editor with custom visual scripting to scalable open-world streaming.

  • Ramen VR - Zenith: The Last City (March 2022): CTO Lauren Frazier explains how ECS helped them scale up their gameplay for managing a VR MMO game architecture.

  • Electric Square - Detonation Racing (March 2022): Lead Developer Jonas De Maeseneer covers the use of ECS for deterministic gameplay for QA and design loops, as well as streaming for a high-pace Apple Arcade racing game.

What’s new since the 1.0 release?
Since hitting release status for Entities, we’ve been keeping a close eye on community feedback in order to improve package stability, resolve bugs, and address high user pain issues. A small set of these improvements are available in the pre-release Entities 1.1 package and can be viewed in our changelog.

There have been other significant updates in the DOTS realm outside of the Entities package:

  • With the new 1.1 version of Unity Physics, we’ve added a collider inspector that shows collider data in detail and allows you to analyze and modify options like collision filters, collider types, geometry, and more in real-time. We’ve also simplified the process for runtime collider modifications with new functions and components that make it easy to create unique colliders and disable the sharing of collider data. Lastly, the dynamic scaling of rigid bodies has been made more intuitive by carrying the uniform scale of a rigid body present at edit-time into its runtime LocalTransform component. You can learn more about these changes and other relevant additions via the Unity Physics changelog.

  • An Editor fix for crashes occurring when using AudioClip.GetData() will be arriving for 2022.3.13f1.

  • After releasing our first experimental version of Burst Occlusion Culling for the 2022 LTS, we conducted many tests and rounds of user feedback. These efforts helped confirm some of our approaches, but we also noted the performance gains in real-world scenarios have not met expectations. We are working on improvements for future versions, but to avoid confusion, we will be removing this experimental feature from general availability in the meantime. Many thanks to those of you who took the time to test the early release of this package. We’ll share details of our new approach in an upcoming roadmap update.

For more details, please check out the detailed changelogs for each of the corresponding packages:

Grateful for Your Feedback
We’ve received a stellar amount of feedback from creators like you over the past months! Know that we are carefully reviewing your input as always for consideration towards our roadmap.

In particular, this past August we hosted the 2023 DOTS Dev Blitz Day — a special event where experts from the DOTS team participate in live discussions and Q&A with users on the Unity Forums and Discord. We received many excellent questions from those who participated and continue to see predominant feedback themes in the areas of animation, open-world creation, physics, and overall DOTS ease of use.

It’s always a gift to hear from you all, be it here on the forums, on social media, or at live events like GDC and Gamescom. Thank you and please continue to keep us posted on your thoughts and needs as you’re working with DOTS.

As always, the best way to ensure your feedback is heard is by submitting and voting on ideas within our public roadmap page. Each card on the page can be clicked on to see details and voting options, and all feedback shared here is directly routed to the proper product teams.

Roadmap
We continue to expect an increasing volume of feedback as creators leverage ECS with Unity 2022 LTS. Our roadmap remains stable and on-track for the coming year as we continue to invest in the following areas:

  • Consolidation of ECS workflows: We’re dedicating a significant amount of resources towards understanding and addressing feedback from users adopting ECS in production. Based on the feedback that we’ve received so far, we’ll be focused on the following:

● We’re acting on opportunities to further consolidate ECS workflows so users have a better time iterating and discovering gameplay in their projects. These consolidations also intend to make working with ECS more familiar and approachable to traditional Unity users experienced with GameObject workflows.

● To support fast iteration for your gameplay, we’re preparing to ship the official production-ready release of the ECS Character Controller package (formerly known as “Rival - DOTS Character Controller” on the Unity Asset Store). You’ll be able to use this package to easily add an efficient and highly-customizable character controller solution for ECS, and it can be used with the Unity Physics or Havok Physics for Unity packages. We also have other controller styles in progress that we’ll share more information on in future roadmap updates.

● We’re developing a unified authoring workflow that will allow you to select your preferred physics backend via the Project Settings. Game productions can be unpredictable and sometimes need a different physics engine than what they started with. The need to change backends shouldn’t occur often, but productions have been finding it cumbersome when it does. With this workflow, you’ll be able to switch to the backend you need without having to redo your project. We’ll share more specifics in a future roadmap update.

● We’re working on a new global entities store. Previously, entities were allocated per world and it was often confusing to determine which world an entity belonged to. Entities will be allocated globally so their values are unique when created in different worlds. When debugging, there’s no ambiguity anymore about which world an entity belongs to and this ensures that the data displayed by debug watches is always reliable.

● In addition, entity versions will always be odd. Even values represent unallocated entities, which makes it more convenient to determine what’s what when debugging the insides of the entity store.

● We will be restoring “Frame Selected” / “Lock View to Selected” (”F” / “Shift+F”) functionality in subscenes.

● Official DOTS support for Unity 2023 is coming soon.

  • Enabling cross-play multiplayer game creation: Multiplayer games continue to be a key focus, and our teams are working hard to provide efficient out-of-the-box support for a variety of multiplayer genres. This includes a netcode unification solution delivering all the features you need, a central location in the Editor for multiplayer tools and services, a Playmode for faster testing of multiplayer games, an upcoming multi-platform crossplay sample, and much more! See the details in our November 2023 Multiplayer Update .

  • Enabling open-world game creation: Scalable open-world authoring and runtime capabilities continue to be important for the Unity editor. This is currently being developed on two main fronts, animation, and worldbuilding. On animation, we continue to progress toward a highly performant, customizable 3D animation system that’s powered by our ECS foundation. For worldbuilding, we’re working on better environment workflows more broadly.

  • Improve parallel scheduling and configuration: In the current version of DOTS, all systems run on the main thread and rely on those systems to schedule jobs to enable parallelism. We’re working to improve the out-of-the-box parallelism of DOTS by enabling systems to run in parallel, whether or not they schedule jobs internally. Additionally, the job system is missing configuration controls needed for adjusting how jobs are run to best enable performance for individual game needs. We’re working on expanding the priority and control of how jobs run to prevent common issues users see, such as unintentionally waiting on long-running jobs on the main thread.

  • Improve Entities Graphics performance: Another initiative we are looking at is the support of DOTS deformation in order to provide a mesh deformation pipeline that is performant and scalable - that works on any platform with GPU compute capability supported in DOTS. We’re currently working on improving the GPU performance, as well as the current workflow, for a better and more seamless integration into the engine/SRPs. The next versions will also bring broader material support and URP support for Motion vectors.

  • New Education Content: Our ECS starter samples provide a strong foundation for new DOTS users to understand and bridge the concepts of the C# Job System and ECS. We’re currently working on expanding this repository of samples into guided Unity Learning pathways that give you the skills necessary to start building your multiplayer project with DOTS. Additionally, we will be introducing a new sample that extends the Megacity Multiplayer sample to showcase how those capabilities can be implemented in a cross-play environment, including mobile devices and with a significantly larger number of concurrent players. More details on this announcement will be communicated at Unite 2023.

Our public roadmap page has been updated to reflect recent changes and what we plan to deliver next.

See you soon at Unite 2023!
Learn more about DOTS at our Unite 2023 conference this November 15-16 in Amsterdam. You’ll be able to hear relevant updates across two key sessions:

You’ll also be able to catch other DOTS-related sessions from our Multiplayer and Physics teams:

Be sure to keep an eye out for the debut of an exciting new sample from the DOTS team! This upcoming URP-based multiplayer sample game features multi-platform crossplay and support for mobile devices, all based on Unity 2022 LTS features.

Let us know if we’ll see you at the event, connecting with devs and creators like you is always our favorite part of Unite! For those not able to join us in person, the Keynote will be available to stream on our YouTube and Twitch channels. You’ll also be able to send in questions during Unite’s virtual Ask The Experts event taking place on Unity Discussions and Discord.

Thank you!
We look forward to seeing the ambitious creations you develop with ECS and connecting with you all again in our next roadmap update. In the meantime, keep us posted on your thoughts and questions here in this thread!

  • Ashley
22 Likes
2

Look forward to seeing how this does in practice. It’ll certainly be interesting to see its limits.

4 Likes
3

From the ECS 1.2 changelog

From my bug report

Since CopyAndReplaceEntitiesFrom is going away and never worked. Will there be a way to deterministically control entity iteration order if you want to roll back a world?

1 Like
4

This really helps, thanks!

Does it mean there’s no OC available for DOTS at the moment or near future till it’s improved? Do we have an understudy or hybrid method to approach OC in DOTS Graphic? Where can I find report in detail about this deprecation?(PS: Just about to test it next month:eyes:)

Another brilliant work! can’t wait.

That’s okay, even it’s more or less the same announcement for Anim and World Creation last year… personally I’d rather expecting Animation Status Update (Q4 2023).

Questions about new upcoming world building tools:

  1. Is there going to be any converting workflow to adapt current Terrain Tools outputs and products to next upcoming new one?

  2. Is it out of beta yet?

enjoy the Unite~

5

Thanks for the update!

I’m raising a flag here regarding determinism. Our project relies on it for multiplayer and having global IDs would probably break it. What workaround do you suggest? Would it be possible to manually specify which store a World uses?

6

The global entity store was designed taking the needs of Netcode for Entities into account, and the way rollback is done there doesn’t rely on stable IDs. Doing a low level rollback by simply swapping the chunks like CopyAndReplaceEntitiesFrom was doing never worked reliably and keeping this function around, even without introducing the global entity store, was a liability.

Now the good news is that entity creation is still deterministic. Chunk order remains deterministic per world. The only thing that shouldn’t be done anymore is to sort stuff based on entity ID. E.g. for parallel ECB needs, you should always prefer using the chunk index.

We want to keep pushing entities forward. I cannot comment at this point on our plans, but the global entity store enables important improvements. Unfortunately, this means that we will need to eventually break a few things. We don’t make those decisions lightly, and we’re sorry for the inconvenience.

That being said, we’d like to hear more about your use cases where you reckon that global entities would interfere. If you prefer to avoid discussing your project on the forum, please reach out to me: fabricel (at) unity3d.com. Don’t forget the “L” after my name in that address.

6 Likes
7

Thanks for the reply.
Our use case is: we don’t use rollback. We use SerializeUtility.SerializeWorld/DeserializeWorld to send the world over to clients, then we just share game inputs during gameplay like RTS games do. (NB: SerializeUtility changes the entity ids + chunks so when a client joins, the server and other clients have to perform the same operation)

You say Entity order is still deterministic, so maybe gameplay won’t break. However we’ll have to fix our “desync detector” that relies on entity IDs being the same to diff. But more importantly, I find it risky to accidentally add new non-determinisic gameplay code if IDs are not (e.g. Sorting by value, like you said).

EDIT: at the moment, we also support Entity references when sending GameInputs to the server (I use my spell on THIS enemy). That will need to change to IDs we manage ourselves for all entities.

2 Likes
8

This is a major concern to me, as I rely on full determinism of test scenes to track performance regressions, and those algorithms require a way to order an arbitrarily-acquired list of Entity references in parallel. Sorting that list ensured determinism, but these changes will break that.

If it is possible to recover a per-world truly deterministic run-to-run ID via EntityStorageInfoLookup, that would likely be sufficient for my use case.

5 Likes
9

Unity’s net code is designed for a generic FPS game, and it is highly specialized for that application. It does not rely on determinism. It uses an authoritative server. By writing your own specialized netcode it is possible to create a tailored experience. I don’t want to be uncharitable, but Unity clearly has an incentive to push its own networking which has fees and is one of the ways Unity generates revenue and an area that Unity management views as one of its main growth drivers. The upper management clearly does not understand that high-end network games require tailored solutions, not this generic garbage.

  • It seems like Lock-step determinism is still possible. But deterministically rolling back / resetting a world to an old state is not possible. Or at least not without having complete control over all systems including default Unity systems.
  • Without deterministically rolling back when you join or rejoin a game everyone will need to rerun the simulation from the start, and you will not be able to combine Lock-step determinism with client-side prediction.
  • Entities relying on memory location to ensure determinism is problematic. This should be abstracted away and not relied on for determining the game state.

The game I am working on was designed for Deterministic roll back with client-side prediction and lag compensation. It sounds like we will have to fall back to old fashion Lock-step determinism and comment out our more advanced features. For testing we already had to disable the broken CopyAndReplaceEntitiesFrom.

I am not sure if the application will need to close and restart every time someone wants to join a new game but I guess yes. Which is bad. I don’t think It is possible to reset to the defaults created by ICustomBootstrap.

2 Likes
10

I’m curious about what this will actually mean in practice.

Right now most of the unity system which do use jobs intensively (physics, animation rigging, HDRP/URP, CPU particles) are still mostly configured to run about at the same time as they would on the main thread (e.g. scheduling and then immediately calling complete on the job handles to make sure their work is done before the next part of the frame timing loop). My guess is that the above text is mostly talking about making it so jobs are scheduled earlier, and then allowed to fully run on the background thread to complete running before their (running state “finish”) handles are “.complete()”-d (release to the main thread) later in the frame?

I ended up making a presentation about this a couple weeks ago,
specifically on how to use the job system without making it blocking on the main thread for anyone interested:
https://docs.google.com/presentation/d/1NOMPJ0EnPqpR0rhwXWJKENZ1yakhHvoiBTHb_-Zwv-c/edit?usp=sharing

Or is this more about changing the underlying dynamic of being able to finally schedule jobs off of the main thread?

Or is this more about just making the individual dots systems which aren’t as tied together through job dependencies, all get schedules at roughly the same time (e.g. update by default) so they just execute in relative parallel before all getting completed in something like late update by default?

2 Likes
11

Thanks for the updates, glad to see consistent information coming through!

I very much appreciate that all of the work you all are doing on ECS is open source. It means that when bugs arise, we can make a small tweak to the package. It’s a night and day difference from working with GameObjects.

I’m interested to hear more about the global entity store. We also rely on entity ID determinism for our regression analysis, however, it would be fine as long as we are able to manually reset the entity store, or have the ability to create a second store. I’m happy to have a global one by default.

12

Thanks for raising this folks - as @Fabrice_Lete mentioned, while we took care to stage out the changes here in order to maintain compatibility with our intended deterministic behavior, clearly there was other behavior we either should have taken care to preserve, or communicated from the start that the behavior never intended to be well defined.

As a public API CopyAndReplaceEntitiesFrom induces a commitment. Having discovered the functionality was incomplete and fragile, however, our options for support became pretty limited. I’m hopeful we can provide support for the use-cases y’all have outlined without recourse to that specific API (or at least better define the behavior).

To that end please take @Fabrice_Lete on his offer or email me directly joe.valenzuela -at- unity3d.com . You’ve spelled out some of the use cases here (great) but honestly I’m not sure I entirely follow what’s going wrong and I’d appreciate more context.

6 Likes
13

Discussing this privately is definitely possible. I just feel a lot of confusion still exists and think it would be helpful to get input from some of the other affected users on this forum.

@msfredb7 is using SerializeWorld world to synchronize his states. I find it somewhat surprising that he is getting deterministic results from that. The whole point of CopyAndReplaceEntitiesFrom was that it “Guarantees that the chunk layout and order of the entities will match exactly.” Depending on what he is doing his results are totally plausible, just probably not in the general case.

A lot of different meanings of determinism exist. The kind that would be the most effective would be where the explicate state is cloneable. Simple clone the ECS data and you can recreate the future game state deterministically. This is more than what CopyAndReplaceEntitiesFrom offered. Because CopyAndReplaceEntitiesFrom is implicitly linked to the chunk layout.

Unity physics, I already believe, supports stateless determinism and a clone function. Uncoupling the ECS data used by the built-in Unity systems from the chunk layout probably has performance implications. I have only spent the bare minimum of time looking through the ECS source code. Someone like @DreamingImLatios would know better than me. Since as far as I can tell Unity ECS does not guarantee cloneable determinism. It remains possible now or in the future that the transform system or build physics world system etc. could have code that embeds chunk layout implicitly into the ECS data. Two examples that come to mind are.

  • An entity job that outputs a list where the order of the returned list can differ depending on the entity iteration order. Obviously, you can sort the list, but this would not be necessary if iteration order was deterministic. I don’t know if any Unity systems suffer from this issue.
  • Cases where the ordering of commands in an entity command buffer causes desynchronization.

Internet upload speeds are still slower than one would like. Affected users have the option of reducing the scale of their game and or slowing the update rate. This may allow switching to a custom client-server. RTS games are notoriously difficult to network because of the complexity of the interactions and the fact that often the entirety of the units will be moving and visible either on screen or on the minimap.

I don’t expect deterministic lockstep and or cloneable determinism to be officially supported anytime soon. I would like that to be the case but honestly, I have not been expecting that for a while. We did spend extra time making sure our code base could support such a design and if it becomes more viable in the future, we can adopt it then. For now, we will likely be using a brute-force delta compression design. Which later can be adapted into allowing resynchronization or rejoining.

3 Likes
14

Thanks for everyone’s feedback!

So, I’m trying to identify our options here. Would the following suggestion from @DreamingImLatios address everyone’s concerns? If not, could you come up with an example of something that was possible before but would not be with that new approach?

As for CopyAndReplaceEntitiesFrom, that function was copying chunks from world to world and relied so much on the chunk layout that it could only ever do the right thing in the simplest of cases. So I’d be curious to know in which context some of you managed to use that function, so we can figure out a way to provide an alternative.

Like I said before, if there are details about your projects that you’d rather not share in a public space, feel free to send me an email at fabricel (at) unity3d.com but if you’re fine otherwise, keeping the discussion public is better.

2 Likes
15

To be honest, I don’t even like my suggestion all that much. Doing those entity lookups into some other data structure would involve lots of random accesses. Personally, I would prefer if a WorldIndex was encoded into the 64 bits of an entity that could be masked out to get a deterministic value (though if it were the most-significant bits, masking wouldn’t even be required for sorting). I suspect someone is going to complain about a cap on NetCode thin clients if that were done though.

4 Likes
16

Sure, what I have in mind would be something that doesn’t require a random access. I was more focused on the functional side of it : a deterministic per-world id per entity.

6 Likes
17

I would want the sortkey to be independent of an global entity ID. Only depending on the creation/destruction order of entities within the world to which they belong. The key should also remain constant, after creation, and only be released after the entity is destroyed.

I am not sure if it would be worth it, but I think it would be possible to make destruction order irrelevant and still maintain determinism. By not recycling/reusing sortkeys immediately and instead waiting until at least after the next update/fixed update. Then sorting the reused keys before reuse. This should keep destruction order from affecting the determinism of pure entities. However, it may not be worth it because entities that use physics probably still need a defined destruction order.

1 Like
18

If the sort key needs to be reused, it would require a similar kind of logic to what the entity store is actually doing. Which means keeping track of all the sort keys in use and allocating them, etc. I’m not sure that’s the right path to take, it sounds unnecessarily expensive.

What if it was instead a simple 64 bits counter that gets incremented for every entity allocated?

Usual disclaimer: I’m just spitballing at this point, don’t take any of this yet as a “here’s what we’re going to do”.

1 Like
19

A 64-bit counter is a lot, and it would work for me. I am not using millions of lightweight entities; so, I would not worry about the counter wrapping around and matching the same entity, if it was still alive, which could be the case for a static entity. I would like some way to set the counter so that it could be reset to an earlier world state. If reusable IDs are used some way to reset both the counter and reused datastore would also be required.

I am actually already using a 32-bit nonreusable counter only for my root networked entities. Because of bandwidth limitations I don’t think a 64-bit counter would ever replace my network ID system. Which is fine but something to think about. In my case even a 16-bit counter with reuse could replace a 64-bit counter. But again, just using the key for sorting would be fine.

1 Like
20

A 64-bit counter that wraps around eventually: Yes! Exactly what I was thinking as well when reading through this thread. That works well until you reach a ridiculous number of entities ;).