I am very new to WebRTC and am finding it very cool. I have a project idea in mind that I need super low latency video stream from one Unity project on PC into another Unity project running on a Quest 2. I’ve managed to build the examples and I can stream a render texture from the PC game into the Quest and see the video in VR !
The only issue I am having is the latency is ~ 1 second:
It appears to be working okay but I really need less than 0.25 second latency for my project idea to work.
Is the render streaming codec lossless? Is there a lossy option? Could the latency be minimized if the Quest somehow dropped old frames and only rendered the most recent frame?
If I use PC for both projects then the video appears perfectly synced in real time, and even over a local network in web browsers the video appears instant.
It’s only in the Quest 2 I notice the latency becomes very high and sometimes video playback is a little choppy.
Is there anything i can do to improve render streaming performance or use faster decoding? I tried reducing the stream resolution to 800x600 but the frame rates in the Quest weren’t affected and remained kinda slow.
I have the Quest connected to gigabit WIFI and the PC directly connected to the router, I don’t think the network is causing the latency.
I know the Quest 2 can stream realtime HD full screen VR over WIFI, I sometime play VR games from my PC over WIFI directly on the Quest and it works phenomenally well.
But Unity’s render streaming is really struggling to keep up.
Is there a faster decoder or more efficient way to stream video on a Quest/Android?
I just tested by explicitly enabling hardware encoding by setting:
WebRTC.WebRTC.Initialize(EncoderType.Hardware);.
Then when I tried to connect the Receiver the Broadcast log showed a error: [impolite-Unity.RenderStreaming.PeerConnection] Failed to set remote offer sdp: Failed to set remote video description send parameters for m-section with mid=‘0’.
Does this mean the hardware codec is not available? Could it be the reason why the Quest video stream looks so delayed? Can I be sure if h264 is being used instead of v8?
When using the WebBrowserInput demo it works with and without Hardware Encoding enabled, And I do believe I can see a slight difference in frame rate/quality when switching between them. My hunch is that the codecs are changing and h264 is working for WebBrowser demo, but I am unsure how to enable h264 codec for the Broadcast and Receiver demo - it always produces the above error.
I am using a 2080TI GPU, I believe it supports HW h264.
