I’m currently in the process of designing a 2D evolution simulation game using Unity. The game involves thousands of creatures that interact with one another in an evolving ecosystem. Each creature in the game is driven by its own neural network, which influences its behavior based on various inputs.
The main Unity application hosts the simulation code for the ecosystem and creatures. However, to manage the computational demands of the neural networks, I want to offload the processing to an external application. The plan is to stream the data from the input layer of each creature’s neural network to this external application for processing, and then stream back the output that will dictate the creature’s behavior in the game.
I’m exploring the best ways to implement this real-time data streaming. Given the scale of the simulation, seamless and continuous data transfer between the Unity application and the external processing application is crucial.
At the moment, I have not decided whether to make it a web game or a standalone application. Therefore, I’m open to solutions that can work on various platforms that Unity supports. I’m also aware that there may be performance considerations with certain platforms, especially when dealing with a simulation of this scale and complexity.
Has anyone attempted something similar before or have any advice on the best methods for real-time data streaming in Unity? Any insights on handling potential performance issues related to neural network processing and data streaming would also be greatly appreciated.
I have no guidance (yet) but many additional questions / points of consideration.
Is this an end-user app, or is the number of users limited? What’s the purpose of the app and its business model? This changes the design and cost factors a lot.
How much data do you estimate the stream to be? You should be able to calculate a rough estimate (ie kbit/s per creature and then consider the worst case ie 10k creatures). For example, if you have 100 bytes per second of traffic per creature and you have 10k of those, you’d have 1 MB/s of traffic.
According to this, what would be your minimum required bandwidth (and latency)? This may already rule out some use-case scenarios such as using the app over-the-air on a mobile device.
What does “streaming” mean here? Is it a permanent streaming connection like realtime video-streaming, or is it a push/poll (post/get, send/receive) transfer that’s event-driven and possibly limited to nearby/“active” creatures? For thousands of creatures the latter is a more feasible approach.
If this neural network is so processing intensive that a state of the art device cannot handle it, have you calculated the costs when simulating the AI on high-end machines (possibly per user ?!) on a service like AWS? For an end-user app it would not be cost-effective unless you run a subscription model.
I’ve done “tens of thousands of agents interacting” before, years ago when computers were much slower. The main thing to consider is that the overheads of Object Oriented approaches can have a significant impact here. Unity has DOTS, which is the first thing I’d check out if I were doing a similar thing in Unity today.
If you’re doing stuff which could benefit from sidestepping some of C#'s overheads (e.g. array access being internally bounds checked by the runtime) then I’d consider writing the logic as a native library, and calling that library to do the heavy lifting. But I’d profile it first to check that it makes a significant difference in your case, and that the difference is bigger than the marshaling overheads.
I’ve not done this part, where data is streamed out of Unity for… some reason? You’ve mentioned it being an external application, but you’ve not mentioned running it on dedicated hardware. Being a separate application won’t magically make it faster. It’s adding significant overhead and complication to an area you’ve identified as “crucial”. Certainly not the direction I’d recommend!
Or make it a “Job”, even if not involving multithreading (by calling .Complete() immediately).
That’s usually easier than an external Dll and you keep platform independence automatically. If you manage to not need to copy data from regular List<> to NativeList or back, it might be even faster than external native which usually will require marshalling.
