Unity Render Streaming Status Update (Q4 2021)

Unity Engine
, ,
1

Hello everyone. This thread is for sharing with you about the development status of Unity Render Streaming, and the plans for 2022. If you have any questions about the 2022 project initiatives, please send them to this thread.

Many users have asked us about our product roadmap. I will try to answer in this thread as much as possible. If you have individual questions, please post another thread on the Unity Forum. If you have a bug report, please send it to GitHub Issues.

Looking back on 2021
In 2021, we mainly dealt with the following contents.

  • Mobile platform support (iOS, Android)
  • Integration with Input System
  • Audio stream support

We have released 2.4.0-exp.4 for the WebRTC package, and 3.1.0-exp.2 for the Unity Render Streaming package. For details on the update, see the changelog of each package.

Contributors
We have received various feedback from many developers. We would like to thank all the developers who have a proven track record of pull requests being brought upstream in 2021.

WebGL platform support
At this moment, WebRTC package does not officially support the WebGL platform, but developers of the community have been developed.

We pay tribute to all the developers who have helped with WebGL support.

Many other people have sent us detailed bug reports and checked for fixes. Thank you again.

Plan for 2022
In 2022, we plan to focus on improving stability and performance. In addition, there are two additional functions: hardware acceleration support for video codecs and metadata support.

If you have any other requests, please reply to this thread.

Stability
We have received many bug reports from developers and are working on them. Regarding the behavior of Unity2021.2 on Apple Silicon, we have confirmed that it is currently unstable. We will deal with it as a priority.

Also, using hardware acceleration encoder issues haven’t been addressed since they were first reported in 2020. We will raise the priority and respond. The related tickets are listed below.

Performance measurement and improvement
We have received reports of delays in video streams compared to Google Chrome. We will continue to investigate and respond to problems.

We continue to improve audio delays with user feedback.

Hardware Acceleration Support
Many users have requested hardware acceleration video codecs.

We have also received requests for hardware decoder support.

In addition, we have received requests for APIs for users to create their own codecs.

We discussed about the topic. I hope we can share our plan in the near future.

Metadata Support
Encoded Transforms API is to add metadata to Video Frame. The draft is published by W3C.

Content related to metadata is also discussed in the Unity Forum. It is used to send information that needs to be synchronized with the video frame. For example, the timestamp on the stream sender and the location information of the camera.

Other Features
We plan to add the following in Unity Render Streaming.

Zero Configuration
To stream a Unity scene which the developer created, component settings are required. Simply add a package and you’ll be able to stream instantly without any configuration.

Command line options
After creating the executable file, it was necessary to rebuild to change the settings such as signaling server URL and port. Providing command line options simplifies configuration when deploying the server.

Controlling the quality and resolution of the video stream
The video streaming couldn’t freely choose the codec and bandwidth. Prepare an API that allows fine control.

Frequently Asked Question
We have received the following questions from users.

  • Hololens2 support
  • WebGL support
  • Validated package
  • Various hardware acceleration of the codec
  • VR / AR platform support

Hololens2 support
Inquiry about the support of the Hololens2 has been asked from many users. It is discussed in this Issue.

Mentioned in this issue, we are currently lowering our priority due to the high cost of supporting Windows ARM64.

WebGL support
Community-based development. You can check the operation by checking out the following pull request source branch.

Landing on the upstream is still difficult at the moment, but we will continue to work towards future support.

Verified Package
We have received questions about when to.

Unity package is promoted to a validated package, it guarantees a 3-year long term support (LTS). Please see the following materials for the package life cycle.

The two packages are based on the package development cycle and are currently in the experimental stage. Development team aims to move to a validated package, but needs to clear challenges for long-term support and is currently working on it.

Support for various hardware codecs
The following hardware codecs are candidates, but the support time is not decided.

  • VAAPI (Support for AMD GPU) Support
  • Windows Media Foundation Codecs
  • Intel Media SDK

VR / AR platform
VR/AR platform is not supported yet, but developers have already reported it. Latency issues have been reported with Oculus Quest 2.

WebRTC package includes an AR Foundation sample, so please give it a try and give us feedback.

In Closing
We hope this allows you to get a better understanding of how we are approaching development. Also, We wait for your feedback in this thread.

Last, best wishes for the holidays and New Year 2022!

1 Like
Unity Render Streaming Status Update (Q4 2022)
2

Thank you for this thorough update. I would like to put it my vote for AMD support. Please include the latest hardware encoding for AMD, it’s a shame to let so many AMD servers go unused.

3

Thank you for your feedback.
I would like to discuss about the priority of supporting HWA codecs.