How do I exactly stream audio to the browser?

Unity Engine
,
1

EDIT: Current situation: I have most of a program working but when users connect two issues happen. 1. the host application stops working/rendering, only the clients see the image now. 2. only a single client appears to be able to interact with UI, the rest are just getting a stream with no way to click on the buttons.

My current suspicion is that the unity input is being “stolen” by the client and thus none of the other tabs work. I’m testing this locally with local host and multiple tabs/windows, as the server we intend to deploy to is headless and a bit of a pain to use. (It’s accessed through my oculus VR Virtual Desktop application because of physical space issues we are having at the moment, which makes iterating there a bit painful.)

The current puzzle is how to get every client watching through the broadcast page to interact with the UI as they wish, instead of locking down to a single user’s input at a time. The app has no controllers or “gameplay”, it is a remote virtual DJ with buttons to interact with it.

Feel free to skip the below, I’m leaving it here cause it’s part of my journey so far, but a lot has been solved between the original post and where I am at. If my situation changes I will also update this post as best I can.

-------- Original post below ---------
I have been trying to get this working, and I can get to my localhost/receiver/index.html page and see the scene if I check the unity setting for “automatic streaming”.

My scene is very simple, just some UI and some sounds (it’s meant to be a live music playing application for our local lan network.)

The video shows up but the audio won’t come through. If I check the stats on the broadcast page, it shows no audio being sent (0kb)

(I can’t seem to highlight the page contents to copy and paste the text as it keeps changing and eats the highlighting, so I’m attaching an image instead)

8982481--1236004--upload_2023-4-30_0-14-32.png
I’m assuming this does not stream audio out of the box? The documentation is all very vague and to be honest I don’t understand what is happening in the demo scene for VideoPlayer Sample (there seems to be too much to the scene for me to understand how the audio is getting sent.) And my examples also seem to be missing an Audio demo scene, at least nothing came up with a search in unity after installing the examples. The default Broadcast scene doesn’t send sound, so I’m guessing there is more to this then just setting automatic broadcast to get sound working. But I’m totally lost because there are SO MANY scripts and most are just from the demo scenes, some don’t even match the documentation names, and more.

Can someone explain to me like I’m 2 years old the shortest, dumbest way to make the audio the main camera hears go out to the broadcast when people view the page?

2

Ok, so I found a work around by copying everything from the Broadcast scene and disabling most of it, eventually breaking it down to just the RenderStreaming object. I reassigned the Streams in Broadcast script using two unity inspectors (one locked on the RenderStreaming, the other opening my camera object.) Somehow this fixes the audio, and it comes through now. Things were cloned, scenes were destroyed, a mess was made, but it’s working.

Total things I did:

  • steal all the objects from the demo Broadcast scene, disabled everything except the RenderStreaming object.
    -turn off Automatic Broadcast in the unity settings, as it was messing with everything.
  • turn ON the Run On Awake checkbox for RenderStreaming > Signaling Manager. Leave everything else the same.
  • Attach new Audio Stream Sender and Video Stream Sender to my own Main Camera.
  • Assign these scripts to the RenderStreaming > Broadcast script Streams property, overwriting slots 1 and 2.
  • Disable the Broadcast Sample script on the same RenderStreaming object, as that just does the UI for the demo we don’t need it.
  • I wasn’t sure what would happen if I removed InputReceiver element from the Streams field, so I just left it there for now.
  • Now audio, video, and the interface appear to be working.

I don’t really understand why this would be different then just running the automatic setup in settings. I would think the automatic setting would have also been doing the audio…it seems that would be more intuitive/useful?

In any case I can move on for now. I’d still love it if someone could explain this step by step what the bare minimum to get a scene + audio transmitting would be. It’s all a bit mysterious to me right now.

EDIT: in stand alone the UI doesn’t work, while in editor test mode it does. Great. I’m going to try adding a custom input receiver to my main camera to copy the setup from the other scene and see if that fixes it.

3

Ok, near as I can tell with opening multiple tabs and seeing which ones update, my setup only allows 1 client at a time to interact with the UI, or in fact do any input at all. Basically the application is acting like whoever logged in first is “the main player host” and the rest are stuck watching without being able to interact, at all. The main application also refuses to render anything until the first client leaves the webpage.

I think this is because they are “taking over” control of the camera/renderings, which prevents the others from doing any input to the game/application. In other words, it’s handling like a multiplayer “log in” system. Since I have just a single EventSystem only the first connection gets control.

What I need instead is for any of them to interact with the buttons in the UI and just trigger the interactions. (The app is a music player with some other UI features meant for anyone in the LAN to control/stream while the application itself runs on a headless server.)

Any ideas how this could be done?

It is a little difficult for me to try different builds because of a couple things:

  1. the editor LIES about controlling the input, it always works in the play Game window during editor mode EVEN if the real exported app would be uncontrollable to the clients. So I have to build an executable with Build and Run to even see the true behaviors.
  2. The server where we would deploy the final app to is headless, and in a part of the house not easily reached. To give some idea of the pain here, I currently use my Oculus headset through Virtual Desktop application to get to the desktop of that machine, as there is no physical room for a keyboard/mouse/monitor nearby. Needless to say this makes trying something out across our live network a wee bit of a pain. So I’ve been testing with localhost, and praying that the network version will behave the same.

I’m certainly open to suggestions. I want to give all clients control over a shared media player experience with some extra unity flair. (custom dj interactive AI with a few other neat tricks).

4

Hi, I think this issue is same with yours, isn’t it?
https://github.com/Unity-Technologies/UnityRenderStreaming/issues/400

5

Sorry for the late reply, been busy on other stuff.

It is not quite the same as my issue, I think. Because that user mentioned multiple UI elements being triggered by other clients. I think from what I read, they have TWO UI, one per user? In any case, my experience is that only a SINGLE client, or if NO clients then the host, can interact with the UI. Everyone else just gets ignored.

In my case:

  • I have a single UI, as if designing an “offline” standard unity application.
  • When starting, the main host can interact with the UI.
  • Once a user “joins” on the stream/site, the main computer cannot interact anymore. The only working UI interaction is on the newly joined stream on that webbrowser.
  • If a second user joins the stream (currently done by just me opening another tab and joining there) > now the second user is the “main controller” of the UI, and BOTH the original host with the executable AND the first client cannot interact with the UI.

It is like the UI is being passed from user to user, and only working for a single user at a time. I’d like to have each user just also be able to push and share all buttons. And of course it would be great if the main executable on the host also still worked, instead of being taken over by the stream user (who may not be me and may not even be at their desk in our case.)

I feel like maybe I’m missing something fundamental about how to setup UI for multiple, potentially infinite number of users?

6

We added it to the task as an improvement plan that allows multiple users to operate the same screen.

1 Like
7

Is there a place I can track when this gets added? I searched github but there are a lot of things to dig through there. A link would be appreciated, thanks!

8

If possible, I would appreciate it if you could raise an issue on GitHub.
Issues on GitHub will be updated when fixed and released, so I think you can trace them.