@wkaj: As in: if Unity would natively support multiple listeners and per-listener effects, how should it sound?
(Warning: big post coming up)
Tools for Managing Multi-Listener Mixing
First go for the naive approach, just superimpose the listening perspectives and send them to the stereo output. This will not work for aurally intense games, but there’s also plenty games for which this should be fine.
Next, introduce extra tools with which you can control the mix better. How you would use these depends on the game you’re making, and how you think it should sound, but there’s some possible common patterns. Of the top of my head:
-
Compressor/Limiter/Sidechaining effects on listeners and stereo outputs. It’s hard enough to keep aurally dense Unity games from clipping, let alone when you add multiple listeners to the mix. A simple compressor at the end of the signal chain wouldn’t be an elegant solution to this problem, but it would be a quick fix. I might try my hand at implementing a Compressor component using OnAudioFilterRead, actually.
-
High Dynamic Range audio. The newer entries in the Battlefield series use this to great effect. Again, it’s no magic bullet that automatically cures your mix. But the HDR paradigm does seem like a powerful tool to help you organize dense and very dynamic mixes. Culling and prioritization are handled elegantly this way, which would no doubt come in handy for splitscreen setups as well. Caveats: DICE use custom software (not sure if FMOD could be juryrigged for this), and demands on audio asset creation become heavier.
-
Allow sound artists to prioritize sounds, and have some kind of mixer that manages sound priority between listeners. If player 2 really, really needs to hear this rocket that’s about to hit him, perhaps you can duck the sound of player 1.
Per-Listener Effects and General Audio Engine Thoughts
Your question also started me musing about hacking together per-listener effects in the MultiAudio package, and whether it can be done:
With the current audio implementation in Unity the options for this plugin aren’t great.
The pessimistic answer
Can’t be done. It’s an inherent limitation to the way Unity audio currently works. Filter components cannot be added in such a way that only one virtual listener hears their effects. No actual sound passes through the virtual listeners, so the only place filters can be added is on the real listener.
The failed answer
Finally, I tried hacking OnAudioFilterRead to get buffers from other audio sources, downmix them, filter them, and play them back. The problem was that OnAudioFilterRead timing didn’t sync up with the buffers gotten from the other sources. That, and even this down-mixing trick did work, the other sources would still be heard.
The unfeasible answer
You could, for each filter added to a virtual listener, add a real filter to each real source that it hears. This would work, but the implementation would be quite inelegant. Instead of using one filter instance to filter all affected source (which could even be optimized by first down-mixing those sources to a single buffer), you have to use many instances that all do exactly the same thing.
It would actually make sense to do this if the current filters were affected by spatial properties (where they are in the scene with regards to audio sources), but they’re simple stereo filters. An exception might be the Reverb Zone effect, but I suspect it just tweaks dry/wet ratio based on the specified falloff. The actual DSP doesn’t need any spatial context to function.
Another problem here is that all the effects in this supposed chain all get stacked on the objects they affect. Consider this example audio signal chain: source->lowpassfilter->distortionfilter->listener->chorusfilter->reverbfilter. If you simulate the following signal chain with this approach you had better be sure the filter objects were added in the right order. Now consider dynamic inserting and removing effects dynamically. Order is important here, but dynamically managing the ordering of components on a game object not something Unity does well, since it didn’t make sense to do so before post processing and audio effects were introduced.
This highlights an important problem: unity game objects and components are not a sufficient metaphor for audio signal chain definition. I realize Unity wants to use the component paradigm for everything, and I’m well aware that there are many parallels between game scene graphs and audio processing graphs. But there’s also many differences.
Sometimes you have tight links between the two (this object should play this sound), but many times you don’t (this filter should effect all gunshot sounds for 3 seconds, starting now.)
Audio functionality should be represented in the scene graph when it where it makes sense, but managed in another way when it doesn’t.
Anyway, this leads to some things Unity could do.
The iffy answer
- Unity audio filters expose a Filter(float[ ] samples, int channels) method.
- Unity audio sources expose expose a version of GetOutputData(float[ ] samples, int channel) that gets you the next synchronous buffer to play. They also get the option to not produce audible sounds themselves. They become a type of Unit Generator.
This way, in an AudioSource’s OnAudioFilterRead you could then call GetOutputData on several other sources, manually down-mix them, then route the stereo result through one filter.
This would work, but it’s hacky
- For design reasons: Unity’s audio engine still has internal audio routing that you can’t touch, but your hijacking it, patching things together in unexpected ways without its knowledge.
- For performance reasons, audio processing is something you would want to do in C++. Not C#.
The ideal answer
Unity implements a native, much more flexible, scriptable audio routing system, and none of this is a problem anymore. 
Audio Scripting could be done through the currently available languages, and graphical editors akin to visual shader editors.
It just occurred to me that the Fabric plugin by Tazman-Audio really highlights how much we desire better mixing control. They essentially implemented fake mixing. The interface acts as if you are mixing, and it does a whole lot of legwork under the hood to simulate the desired effect, but it doesn’t actually downmix.
Anyway, thank you so much for listening. I’m really curious to hear the thoughts you Unity guys have on all these things.
Oh by the way, here’s another thread I started with some grand speculation on game audio technology and workflow: Singularity - A flexible general purpose audio engine
