I recently started using Unity Render Streaming and I want to share some thoughts and findings because on the Github issue page and here in the forum a lot of issues/questions are about similar topics and are posted again and again. Maybe my code changes (see next post) will help in one case or another.
First off, I’m really happy about Render Streaming. Having this functionality is really cool and opens up new use cases for us. It’s already working pretty well even though some key features for real world use are still missing.
Especially improved and more stable hardware encoding because using software encoding for a high-resolution application running with 60fps is not really feasible. And not so much of a feature but a guide/sample on how to integrate an SFU/media server would also be necessary. Since we don’t have experience with that it would probably take us a lot of time to achieve that.
What also took quite some time (more than feels necessary) was to find out about issues and workarounds for common problems. After reading the docs of the WebRTC and Render Streaming packages I had to read through all the Github issues and the forum as well to look for certain information. What was especially troublesome was to find the right information on how to improve the quality of the video stream. What is a bit annoying since video streaming is what it’s all about. And there are quite a few Github issues about bad video quality. There is a short explanation about bitrate in the WebRTC docs but this should definitely be shown somewhere in the samples of the Render Streaming package. Because out of the box the hardware encoded stream looks pretty bad compared to the software encoded one.
So it would be good to know what the default parameters for the encoders are and which one can be changed and how. The fact that information and issues are split up between the WebRTC and Render Streaming packages makes it even a bit more difficult to find information. Maybe they could link to each other more often in certain topics.
So I want to share some changes/improvements I made to the WebRTC and Render Streaming code that worked for us. I used the source code of Render Streaming 3.1.0-exp.1 and WebRTC 2.4.0-exp.3 and the WebBrowserInputSample with the built-in render pipeline in Unity 2020.3.10f1:
Expose VideoStreamBase.StreamingSize to set RenderTexture resolution via code
Use VideoStreamBase.ChangeVideoParameters() to set encoding bitrate (and also downscale resolution when using software encoding, see WebBrowserInputSample.HandleStreamStarted())
Free resources of video/audio tracks when a receiver/web browser is disconnected (see Github issue, Broadcast.Disconnect())
Fallback to software encoding when hardware encoding is not possible on startup (see WebBrowserInputSample.StartRenderStreaming())
Free all resources (render textures) when no one is receiving the stream anymore (see WebBrowserInputSample.HandleStreamStopped(), made CameraStreamer/StreamSourceBase disposable)
Specify encoding resolution different from render texture resolution (hardware and software encoding, see EncodingResolution in WebBrowserInputSample and CameraStreamer, creates destination RenderTexture in VideoStreamTrack with different resolution than the camera’s RenderTexture)
Feel free to use the files as they are or copy the code you need. Be aware that if you replace a file that you might get compilation errors because of possible API changes I made. Also I can only give you limited (if any) support if my changes don’t work for you/your scenario.
Everything was ok in the unity editor but when I release PC ， I start to connect it and the picture quality is very poor and after 30 seconds later everything is fine 3.1.0-exp.1 and WebRTC 2.4.0-exp.3 @three10 @ kazuki_unity729
Everything was ok in the unity editor but when I release PC ， I start to connect it and the picture quality is very poor and after 30 seconds later everything is fine 3.1.0-exp.1 and WebRTC 2.4.0-exp.3 @three10 @ kazuki_unity729
@chrisming999 Yes, I can confirm that when first receiving the stream it takes a few seconds before the quality of the stream improves. but for me it is really only about 3 seconds and not 30.
@three10 Thanks for your sharing. I am very new to Unity and this Unity Render Streaming. I downloaded the same versions of these two packages. Could you please give me some instructions on how to use your files? Should I add these files to the RenderStreaming gameobject to replace the original files and keep the renderstreaming.cs? What about the videostreambase.cs, videostreamtrack.cs and StreamSourceBase.cs? I did not find them :(.
Maybe the author has a different way to use those files, but this is how I did it:
Copy the com.unity.renderstreaming@3.1.0-exp.1 and the com.unity.webrtc@2.4.0-exp.3 folders from / Library / PackageCache, then paste them into another directory
Replace some of the files inside those cloned packages with the ones the author provided
Put the cloned packages into / Packages
In your project’s Package Manager, click the (+) plus button on the top left corner and select Add Package from Disk
The resolution and the data rate will be exposed in the Inspector, through the altered WebBrowserInputSample.cs script, in the WebBrowserInputSample scene
We can set the resolution, but the frame rate is still not good. I have a heavy scene with around 30 fps on a desktop PC and it will drop to around 12 fps when broadcasting it with Unity Render Streaming. I hope the performance gets better in the next version.
Thank you so much for your kind reply! @liquify I really appreciate it.
I followed your instructions and copied the two packages in a different directory. Then I used the first 5 files provided by the author to replace the files in the com.unity.renderstreaming@3.1.0-exp.1/runtime/scripts and the last 1 for the com.unity.webrtc@2.4.0 exp.3/ runtime/scripts.
Then I created a new project. When I did the step3, I got an error in the WebBrowserInputSample.cs file:
CS0246: The type or namespace name ‘CopyTransform’ could not be found (are you missing a using directive or an assembly reference?).
I tried to skip step3 to do step4. I still got the same result.
Could you please tell me did I replace the files in a right way? Is step3 necessary?
Try to replace the WebBrowserInputSample.cs in the cloned package with the original one and replace the one in / Assets / Unity Render Streaming / 3.1.0-exp.1 / Example / WebBrowserInput folder instead. If the customized packages are shown in the Package Manager (see the screenshot below), step 3 and 4 won’t be necessary:
If the errors persist:
Close the project
Delete the Library and the obj folders from the project
Delete all folders from from C:\Users<Your Username>\AppData\Local\Unity\cache\packages\packages.unity.com
Thanks a lot! It finally worked. Though when I tried to change the bitrate in the play mode, nothing got changed (observed from Chrome webrtc-internals). I guess there is still a lot of bugs and problems needed to be solved.
Sorry @xiaolongyin , I haven’t looked in here for a while. Thanks @liquify for helping out!
The way I imported the packages was by downloading the source code directly from the Github release pages, e.g. Releases · Unity-Technologies/UnityRenderStreaming · GitHub, and copying the necessary folders into the Unity project. So I didn’t use the Package Manager at all for this.
The ‘CopyTransform’ script is part of the Render Streaming Samples and not in the main packages, so make sure to import/copy them as well. E.g. from the Zip from the repository you could just copy the whole ‘UnityRenderStreaming-3.1.0-exp.1\com.unity.renderstreaming’ folder.
But as you already got it to work this info comes a bit late.
Changing the bitrate of the stream works for me. Make sure to set it before starting the application and not afterwards. Also try debugging and setting a breakpoint at the position where the bitrate is set to make sure it is actually called. As @liquify already mentioned setting the frame rate doesn’t work because it is not implemented by the package. Frame rate is basically just the same as the application’s FPS. To change the streaming FPS probably a separate update loop for the stream independent of the application’s update loop would have to be implemented.
(observed from Chrome webrtc-internals). I guess there is still a lot of bugs and problems needed to be solved.