How to Record In-game audio and microphone audio together?

Hi!

I have a video playing in unity and I enabled the microphone in unity.
I need to record both video and microphone audio together.
I can't find a way to record both video's audio and microphone's audio together.

I am using AVPro video for playing video and I am using Microphone. Start to use the microphone.
I tried it with NatCorder and unmuted microphone audio. And recorded AudioListener of the scene.
But when I open the recorded file, I can hear microphone audio only but microphone audio was not accurate(maybe lagging) or no audio at all
It looks like Audio Listener can't listen AVPro Video's audio playback.
If someone has a solution, please let me know
Thanks.

1 Like

[quote=“reysantana”, post:1, topic: 831343]
Hi!

I have a video playing in unity and I enabled the microphone in unity.
I need to record both video and microphone audio together.
I can’t find a way to record both video’s audio and microphone’s audio together.

I am using AVPro video for playing video and I am using Microphone. Start to use the microphone.
I tried it with NatCorder and unmuted microphone audio. And recorded AudioListener of the scene.
But when I open the recorded file, I can hear microphone audio only but microphone audio was not accurate(maybe lagging) or no audio at all
It looks like Audio Listener can’t listen AVPro Video’s audio playback.
If someone has a solution, please let me know
Thanks.
[/quote]

Stuck with this same issue here, did you find a way to do this?

[quote=“ROBYER1_1”, post:2, topic: 831343]
Stuck with this same issue here, did you find a way to do this?
[/quote]
One way to do it is to mix Microphone and Game sounds by hand while recording.

To record in-game sounds, attach a monobehavior with “OnAudioFilterRead()” to the audio source you want to record. Supposedly it also should work on AudioListeners, though I haven’t tried t hat.

Then mix the sound you receive with the microphone sound and feed it into recorder.

Be aware that audio programming in unity is not the most trivial thing to do, as it is done asynchronously in a different thread.

One other problem is that unity records microphone audio to an audio clip, which is idiotic.
But that may be solveable with a 3rd party asset.

Can you pull the audio signal from the video? If so you can mix them. Beware, you’ll need to make sure you have the same sample rates and you handle mono or stereo appropriately.

[quote=“neginfinity”, post:3, topic: 831343]
One other problem is that unity records microphone audio to an audio clip, which is idiotic.
[/quote]
Why is this a problem? You can get the audio with GetData.

2 Likes

[quote=“EternalAmbiguity”, post:4, topic: 831343]
Why is this a problem? You can get the audio with GetData.
[/quote]
You will not be able to record continuously, and periodically you’ll need to refresh the clip, which will interrupt the stream and likely create clicks. It will also create issues trying to sync things.
GetData also does not give you precise control over which samples you want to pull.

A better option would be to have an equivalent for OnAudioFilterRead for pulling mic data. At least as an alternative option. Or some stream-like interface with an internal buffer.

2 Likes

It doesn't have to be a perfect match, this is in an app that is a video chat call where we simply want to record what is on screen + both the local user mic and the incoming video call audio (with Agora).
It seems way harder than I imagined to record that all together within Unity - we are using Natrecorder to record the video and currently one input which in our case is the local microphone which sort of picks up a bit of the in local audio feed but not enough.

It doesn't have to be perfect we just want to combine the local microphone audio stream with the local unity game sound or another audio stream which can be recorded with the local video feed

[quote=“ROBYER1_1”, post:6, topic: 831343]
It doesn’t have to be a perfect match, this is in an app that is a video chat call where we simply want to record what is on screen + both the local user mic and the incoming video call audio (with Agora).
It seems way harder than I imagined to record that all together within Unity - we are using Natrecorder to record the video and currently one input which in our case is the local microphone which sort of picks up a bit of the in local audio feed but not enough.

It doesn’t have to be perfect we just want to combine the local microphone audio stream with the local unity game sound or another audio stream which can be recorded with the local video feed
[/quote]

You might consider NAudio. https://github.com/naudio/NAudio/blob/master/Docs/WasapiLoopbackCapture.md

I’ve never tried it from within Unity but otherwise, with loopback you should be able to get almost everything, and you can also record audio from the mic and store/merge it.

[quote=“ROBYER1_1”, post:6, topic: 831343]
It doesn’t have to be a perfect match, this is in an app that is a video chat call where we simply want to record what is on screen + both the local user mic and the incoming video call audio (with Agora).
It seems way harder than I imagined to record that all together within Unity - we are using Natrecorder to record the video and currently one input which in our case is the local microphone which sort of picks up a bit of the in local audio feed but not enough.

It doesn’t have to be perfect we just want to combine the local microphone audio stream with the local unity game sound or another audio stream which can be recorded with the local video feed
[/quote]
Normally, a video code would have some sort of input you can feed framebuffer and audio data in. So, instead of feeding scene sound, you could mix your own and feed it in.
That would require hacking natrecorder, if possible, or writing your own solution. For example, likely it shouldn’t be too difficult to code a theora screen/audio grabber, although the video format is very rare these days. Obviously those would require programming skills.

Another option is to talk to NatRecorder’s developer and see if it has the functionality you want.

I just got this working to record both loopback and the mic:

using NAudio.CoreAudioApi;
using NAudio.Wave;
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Threading;

namespace LoopbackWithMic
{
    class Program
    {
        static void Main(string[] args)
        {

            var outputFolder = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.Desktop), "NAudio");
            Directory.CreateDirectory(outputFolder);
            var outputFilePath = Path.Combine(outputFolder, "recorded.wav");
            var enumerator = new MMDeviceEnumerator();
            var output = enumerator.GetDefaultAudioEndpoint(DataFlow.Render, Role.Console);
            var output_false = enumerator.EnumerateAudioEndPoints(DataFlow.Render, DeviceState.Active).Where(x => x != output).FirstOrDefault();
            var input = enumerator.GetDefaultAudioEndpoint(DataFlow.Capture, Role.Console);
            var capture_a = new WasapiLoopbackCapture(output);
            var capture_b = new WasapiCapture(input);
            List<IWaveProvider> pp = new List<IWaveProvider>();
            var provider_a = new BufferedWaveProvider(capture_a.WaveFormat);
            if(capture_a.WaveFormat.Channels > 1)
            {
                var mon = new NAudio.Wave.SampleProviders.StereoToMonoSampleProvider(provider_a.ToSampleProvider());
                pp.Add(mon.ToWaveProvider());
            }
            else
            {
                pp.Add(provider_a);
            }
            var provider_b = new BufferedWaveProvider(capture_b.WaveFormat);
            if (capture_b.WaveFormat.Channels > 1)
            {
                var mon = new NAudio.Wave.SampleProviders.StereoToMonoSampleProvider(provider_b.ToSampleProvider());
                pp.Add(mon.ToWaveProvider());
            }
            else
            {
                pp.Add(provider_b);
            }
            MixingWaveProvider32 mix = new MixingWaveProvider32(pp);
            WaveRecorder record = new WaveRecorder(mix, outputFilePath);
            var speaker = new WasapiOut(output_false, AudioClientShareMode.Shared, true, 15);
            speaker.Init(record);
            List<byte> bytes = new List<byte>();
            capture_a.DataAvailable += (s, a) =>
            {
                provider_a.AddSamples(a.Buffer, 0, a.BytesRecorded);
            };
            capture_b.DataAvailable += (s, a) =>
            {
                provider_b.AddSamples(a.Buffer, 0, a.BytesRecorded);
            };

            capture_a.StartRecording();
            capture_b.StartRecording();
            speaker.Play();
            DateTime now = DateTime.Now;
            while ((DateTime.Now - now).TotalSeconds < 15)
            {
                Thread.Sleep(50);
            }
            speaker.Stop();
            capture_a.StopRecording();
            capture_b.StopRecording();
            record.Dispose();
            record = null;
        }
    }
}

This would ONLY work in situations where you have an active output that doesn't go to your regular speakers, so maybe not what you want (I didn't see anything in NAudio to create a virtual output).

Edit - found this (https://vb-audio.com/Cable/index.htm), which you might be able to use.

[quote=“EternalAmbiguity”, post:9, topic: 831343]
I just got this working to record both loopback and the mic:

using NAudio.CoreAudioApi;
using NAudio.Wave;
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Threading;

namespace LoopbackWithMic
{
    class Program
    {
        static void Main(string[] args)
        {

            var outputFolder = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.Desktop), "NAudio");
            Directory.CreateDirectory(outputFolder);
            var outputFilePath = Path.Combine(outputFolder, "recorded.wav");
            var enumerator = new MMDeviceEnumerator();
            var output = enumerator.GetDefaultAudioEndpoint(DataFlow.Render, Role.Console);
            var output_false = enumerator.EnumerateAudioEndPoints(DataFlow.Render, DeviceState.Active).Where(x => x != output).FirstOrDefault();
            var input = enumerator.GetDefaultAudioEndpoint(DataFlow.Capture, Role.Console);
            var capture_a = new WasapiLoopbackCapture(output);
            var capture_b = new WasapiCapture(input);
            List<IWaveProvider> pp = new List<IWaveProvider>();
            var provider_a = new BufferedWaveProvider(capture_a.WaveFormat);
            if(capture_a.WaveFormat.Channels > 1)
            {
                var mon = new NAudio.Wave.SampleProviders.StereoToMonoSampleProvider(provider_a.ToSampleProvider());
                pp.Add(mon.ToWaveProvider());
            }
            else
            {
                pp.Add(provider_a);
            }
            var provider_b = new BufferedWaveProvider(capture_b.WaveFormat);
            if (capture_b.WaveFormat.Channels > 1)
            {
                var mon = new NAudio.Wave.SampleProviders.StereoToMonoSampleProvider(provider_b.ToSampleProvider());
                pp.Add(mon.ToWaveProvider());
            }
            else
            {
                pp.Add(provider_b);
            }
            MixingWaveProvider32 mix = new MixingWaveProvider32(pp);
            WaveRecorder record = new WaveRecorder(mix, outputFilePath);
            var speaker = new WasapiOut(output_false, AudioClientShareMode.Shared, true, 15);
            speaker.Init(record);
            List<byte> bytes = new List<byte>();
            capture_a.DataAvailable += (s, a) =>
            {
                provider_a.AddSamples(a.Buffer, 0, a.BytesRecorded);
            };
            capture_b.DataAvailable += (s, a) =>
            {
                provider_b.AddSamples(a.Buffer, 0, a.BytesRecorded);
            };

            capture_a.StartRecording();
            capture_b.StartRecording();
            speaker.Play();
            DateTime now = DateTime.Now;
            while ((DateTime.Now - now).TotalSeconds < 15)
            {
                Thread.Sleep(50);
            }
            speaker.Stop();
            capture_a.StopRecording();
            capture_b.StopRecording();
            record.Dispose();
            record = null;
        }
    }
}

This would ONLY work in situations where you have an active output that doesn’t go to your regular speakers, so maybe not what you want (I didn’t see anything in NAudio to create a virtual output).

Edit - found this (https://vb-audio.com/Cable/index.htm), which you might be able to use.
[/quote]

This looks promising but is it for Windows or desktop only? I forgot to mention our application is on IOS/Android as well as Desktop (windows currently and soon Mac)

Edit: looks like it definitely is Windows API limited

[quote=“neginfinity”, post:8, topic: 831343]
Normally, a video code would have some sort of input you can feed framebuffer and audio data in. So, instead of feeding scene sound, you could mix your own and feed it in.
That would require hacking natrecorder, if possible, or writing your own solution. For example, likely it shouldn’t be too difficult to code a theora screen/audio grabber, although the video format is very rare these days. Obviously those would require programming skills.

Another option is to talk to NatRecorder’s developer and see if it has the functionality you want.
[/quote]

I’m in talks with the Natcorder dev about this, they had a (since removed) piece called the MixerDevice which allowed capture of multiple inputs mixed together ie. game and mic sound. https://discussions.unity.com/t/683160 page-77#post-7257713

I’m testing out the MixerDevice today and trying to identify an IOS crash we had with it when I last tried it.

The creator of next Gen Recorder who also seems to work at Unity had his own extra tool (which all the links for are dead) which records directly the audio going through the audio mixer while combining it with the microphone audio (which is unheard by the local user). I have also reached out to that developer as they claimed the links were still live so I’ll try a VPN or something silly to grab it but I think they removed it from the web for some reason and forgot.
https://discussions.unity.com/t/779645/50

[quote=“neginfinity”, post:5, topic: 831343]
You will not be able to record continuously
[/quote]
A quick search for realtime playback of the microphone came up with the following link.

https://support.unity.com/hc/en-us/articles/206485253-How-do-I-get-Unity-to-playback-a-Microphone-input-in-real-time-

Which has this code that surprisingly works and without that much latency either.

using UnityEngine;

public class MicrophoneHandler : MonoBehaviour
{
    void Start()
    {
        AudioSource source = GetComponent<AudioSource>();
        source.clip = Microphone.Start(null, true, 1, 44100);
        source.loop = true;
        while (!(Microphone.GetPosition(null) > 0)) {}
        source.Play();
    }
}

[quote=“Ryiah”, post:12, topic: 831343]
A quick search for realtime playback of the microphone came up with the following link.

https://support.unity.com/hc/en-us/articles/206485253-How-do-I-get-Unity-to-playback-a-Microphone-input-in-real-time-

Which has this code that surprisingly works and without that much latency either.

using UnityEngine;

public class MicrophoneHandler : MonoBehaviour
{
    void Start()
    {
        AudioSource source = GetComponent<AudioSource>();
        source.clip = Microphone.Start(null, true, 1, 44100);
        source.loop = true;
        while (!(Microphone.GetPosition(null) > 0)) {}
        source.Play();
    }
}

[/quote]

Using that here too it was quite simple code to figure out, just about to test if natcorder’s mixingdevice will work okay for recording on IOS as when I last tried to use it, it had a crash bug

I am pleased to announce I cracked how to do it, it was actually very simple in the end. Make sure you are using Natcorder, use an older version that still has the MixerDevice.cs, make sure to incorporate the fix from here if you are on IOS: https://discussions.unity.com/t/683160 page-69#post-6493429

[Additional fix]
Further searching fixed an issue with iOS. https://discussions.unity.com/t/609451 page-50
Destination array was not long enough. Check destIndex and length, and the array's lower bounds
Edit MixerDevice.cs to increase buffer size .
var copyBuffer = new float[16384];

Then use the same code as them (see link above or use below)

var query = new MediaDeviceQuery(MediaDeviceQuery.Criteria.AudioDevice);
        var microphone = query.currentDevice as IAudioDevice;
        audioDevice = new MixerDevice(microphone, Camera.main.gameObject.GetComponent<AudioListener>());
        Debug.Log("SampleRate: Device: "+audioDevice.sampleRate);
        Debug.Log("SampleRate: AudioSettings: "+AudioSettings.outputSampleRate);
        // Recorder setyop
        var clock = new RealtimeClock();
        recorder = new MP4Recorder((int) prod.x, (int) prod.y, 30, AudioSettings.outputSampleRate, 2);
        cameraInput = new CameraInput(recorder, clock, Camera.main);
        /// Game + Mic Audio
        //_audioInput = new AudioInput(recorder, clock, Camera.main.gameObject.GetComponent<AudioListener>());
        audioDevice.StartRunning((sampleBuffer, timestamp) => recorder.CommitSamples(sampleBuffer, clock.timestamp));

Now you are able to record the screen, the device microphone and any game audio received by the audiolistener in your scene!

1 Like

One thing:
[quote=“Ryiah”, post:12, topic: 831343]

        while (!(Microphone.GetPosition(null) > 0)) {}

[/quote]
This is actually going to block the main thread until some data has been written by the microphone.

For example, OnAudioFilterRead is called at roughly 50fps every time, here something similar could happen.

1 Like

[quote=“ROBYER1_1”, post:14, topic: 831343]
I am pleased to announce I cracked how to do it, it was actually very simple in the end. Make sure you are using Natcorder, use an older version that still has the MixerDevice.cs, make sure to incorporate the fix from here if you are on IOS: https://discussions.unity.com/t/683160 page-69#post-6493429

[Additional fix]
Further searching fixed an issue with iOS. https://discussions.unity.com/t/609451 page-50
Destination array was not long enough. Check destIndex and length, and the array’s lower bounds
Edit MixerDevice.cs to increase buffer size .
var copyBuffer = new float[16384];

Then use the same code as them (see link above or use below)

var query = new MediaDeviceQuery(MediaDeviceQuery.Criteria.AudioDevice);
        var microphone = query.currentDevice as IAudioDevice;
        audioDevice = new MixerDevice(microphone, Camera.main.gameObject.GetComponent<AudioListener>());
        Debug.Log("SampleRate: Device: "+audioDevice.sampleRate);
        Debug.Log("SampleRate: AudioSettings: "+AudioSettings.outputSampleRate);
        // Recorder setyop
        var clock = new RealtimeClock();
        recorder = new MP4Recorder((int) prod.x, (int) prod.y, 30, AudioSettings.outputSampleRate, 2);
        cameraInput = new CameraInput(recorder, clock, Camera.main);
        /// Game + Mic Audio
        //_audioInput = new AudioInput(recorder, clock, Camera.main.gameObject.GetComponent<AudioListener>());
        audioDevice.StartRunning((sampleBuffer, timestamp) => recorder.CommitSamples(sampleBuffer, clock.timestamp));

Now you are able to record the screen, the device microphone and any game audio received by the audiolistener in your scene!
[/quote]

Just wanted to add this only works in some cases, in some cases we would still get an error with recording where the copybuffer in MixerDevice is too slow to fill or overflowing which I am looking into. Absolute headache.

1 Like

I am working with the Natcorder dev on an audio mixed recording solution which I have nearly cracked now - I have both the game mic and game sound mixing together successfully just need to sort some stuff around saving it properly with Natcam which hopefully the dev will look into soon.

I will post the fixed solution back here and on the Natcorder main thread when it's done

1 Like

[quote=“ROBYER1_1”, post:14, topic: 831343]
I am pleased to announce I cracked how to do it, it was actually very simple in the end. Make sure you are using Natcorder, use an older version that still has the MixerDevice.cs, make sure to incorporate the fix from here if you are on IOS: https://discussions.unity.com/t/683160 page-69#post-6493429

[Additional fix]
Further searching fixed an issue with iOS. https://discussions.unity.com/t/609451 page-50
Destination array was not long enough. Check destIndex and length, and the array’s lower bounds
Edit MixerDevice.cs to increase buffer size .
var copyBuffer = new float[16384];

Then use the same code as them (see link above or use below)

var query = new MediaDeviceQuery(MediaDeviceQuery.Criteria.AudioDevice);
        var microphone = query.currentDevice as IAudioDevice;
        audioDevice = new MixerDevice(microphone, Camera.main.gameObject.GetComponent<AudioListener>());
        Debug.Log("SampleRate: Device: "+audioDevice.sampleRate);
        Debug.Log("SampleRate: AudioSettings: "+AudioSettings.outputSampleRate);
        // Recorder setyop
        var clock = new RealtimeClock();
        recorder = new MP4Recorder((int) prod.x, (int) prod.y, 30, AudioSettings.outputSampleRate, 2);
        cameraInput = new CameraInput(recorder, clock, Camera.main);
        /// Game + Mic Audio
        //_audioInput = new AudioInput(recorder, clock, Camera.main.gameObject.GetComponent<AudioListener>());
        audioDevice.StartRunning((sampleBuffer, timestamp) => recorder.CommitSamples(sampleBuffer, clock.timestamp));

Now you are able to record the screen, the device microphone and any game audio received by the audiolistener in your scene!
[/quote]

Hi, i hate to intrude, but would this work on Android & MacOS devices or is it windows specific?

[quote=“Rose932”, post:18, topic: 831343]
Hi, i hate to intrude, but would this work on Android & MacOS devices or is it windows specific?
[/quote]
I don’t know. But if you want to record screen on PC, you can use Joyoshare VidiKit or OBS. They are very useful fo you to record all activities.

[quote=“ROBYER1_1”, post:14, topic: 831343]
I am pleased to announce I cracked how to do it, it was actually very simple in the end. Make sure you are using Natcorder, use an older version that still has the MixerDevice.cs, make sure to incorporate the fix from here if you are on IOS: https://discussions.unity.com/t/683160 page-69#post-6493429

[Additional fix]
Further searching fixed an issue with iOS. https://discussions.unity.com/t/609451 page-50
Destination array was not long enough. Check destIndex and length, and the array’s lower bounds
Edit MixerDevice.cs to increase buffer size .
var copyBuffer = new float[16384];

Then use the same code as them (see link above or use below)

var query = new MediaDeviceQuery(MediaDeviceQuery.Criteria.AudioDevice);
        var microphone = query.currentDevice as IAudioDevice;
        audioDevice = new MixerDevice(microphone, Camera.main.gameObject.GetComponent<AudioListener>());
        Debug.Log("SampleRate: Device: "+audioDevice.sampleRate);
        Debug.Log("SampleRate: AudioSettings: "+AudioSettings.outputSampleRate);
        // Recorder setyop
        var clock = new RealtimeClock();
        recorder = new MP4Recorder((int) prod.x, (int) prod.y, 30, AudioSettings.outputSampleRate, 2);
        cameraInput = new CameraInput(recorder, clock, Camera.main);
        /// Game + Mic Audio
        //_audioInput = new AudioInput(recorder, clock, Camera.main.gameObject.GetComponent<AudioListener>());
        audioDevice.StartRunning((sampleBuffer, timestamp) => recorder.CommitSamples(sampleBuffer, clock.timestamp));

Now you are able to record the screen, the device microphone and any game audio received by the audiolistener in your scene!
[/quote]

Hi Robyer1,

Do you know where I can find the older NatCorder/NatDevice version ? I saw NatDevice removed MixerDevice class in the 1.1.0 version (I guess 1.0.2 should be ok, but I don’t know where to get it …).

Thanks in advance …