Multiple cameras (48+) to color byte arrays.

Unity Engine
,
1

Hi !

I’m trying to use unity as a simulation tools. I need to use a lot of cameras and by a lot i mean something like 48+, the more i can have the better. For now this is on a single machine.Every render gonna be sent over network in a local environment.

Each camera render at a low resolution like 640x360 and my goal is to do 3-5 fps. (This helps a lot.)

Since last week, I’ve been working and testing some stuff. I was able to understand the basics of going from camera.Render to RenderTexture to Texture2D to GetRawTextureData. I use a coroutine to split the workload on more game frame which should run at 30 fps or more (Main camera), we have a ~200ms of frame time for the cameras so it’s a must.

One of the big challenge was Texture2D.ReadPixels, it’s slow since it force a Gpu-Cpu sync… Can’t we bypass that some how ? Can’t i just retrieve data from de RenderTexture.colorBuffer ? Even then, there is no magic, and there is going to be a sync anyway. How could i optimize that for multiple Texture2D ? For now i read them in sequence, most of the time it’s the first one taking a lot of time, others are pretty fast, which make sense… Could i try to use 1 big Texture2D and tiled it for each camera ? But this way i can’t mitigate the sync overhead over multiple frames…

I’ve been able to get something acceptable running in the editor at 4-6 fps with 48 cameras on an empty scene. But it doesn’t include the “last” part GetRawTextureData. Now the bottleneck is the GC with 32MB Alloc which force a Collect that can take 20-40+ ms… Is there a way to pass a byte array so i can reuse it ? Obviously it’s not in the doc, but i don’t understand why you can’t do that ! It would help a lot with this kind of work…

I found this article, it talks about how they solved the same problem of GetPixels. But they don’t explain how they did it, just that they were able to run de GC in a background thread… I tried it but maybe my implementation is terrible cause it doesn’t help at all. Since i have a “huge” amount of memory to take, i think the GC is called multiple times in the same compute frame. If i try the same approach and mitigate the impact over multiple frames, it makes my compute frame time go too high…

So what do you guys suggest me to do ? I would like to stay away from native C++ code, but if it’s the best solution, i can look into it.

(Btw idk if i’m in the right section of the forum :/)

2

maybe this feature would help?

2 Likes
3

Maybe a native plugin sounds like a good idea.
You could use GetNativeTexturePtr to get access to the content of the rendertexture , then from the native plugin side run it in a seperate or multiple separate threads to get the Rendertexture content in the correct format and out over the network.

4

@mgear This looks promising, but doesn’t it need gpu support ? It sounds like async compute… I have a GTX 970 which doesn’t really support async compute (i think). Gonna test it later for sure.

@fffMalzbier Yeah that was my guess too. Do you have examples of how to do that ?

Yesterday i tried the tile approach. Since my gpu support texture size up to 16384x16384, I tested it with a 7x7 grid. Each camera render to a RT (RenderTexture) of 640x360 and when i’m done with the renders, i use Graphics.CopyTexture into a bigger RT (4480x2520) so each camera can have it’s own cell in the grid. After it’s the same thing with Texture2D, but it’s a much bigger.

I wanted to see if the ReadPixels would be faster on 1 big texture vs multiple small ones, but it doesn’t look like it. Processing time is huge like +200 ms. I tried to analyse gpu memory transfert with GPU-Z but i dont see much, so i’m not sure if i got the wrong tool or if i’m not looking at the right thing. Since it’s only 1 call i can’t mitigate it over multiple frame (maybe with AsyncRequest)

— Update

Alirght @mgear , I was able to work with AsyncGPUReadBack and it’s very good ! oO
I’m able to do 10 or 15 fps with those 48 cameras (Main cam lock at 30). Basically 1 frame i queue up requests and in general i get all my data back the next 1 or 2 frames later. So yeah thanks !

I might post some code or a git repo later if others want to replicate that, we never know :slight_smile:

Thanks !

1 Like