How to create AudioStream from byte Array in wave format

Audio & Video
,
1

Hi,

My Unity Version is 2020.3.40f

I am using the ROS2 audio_common package to send AudioData to Unity.

There I continuously receive a byte Array out of which I want to generate an audio stream inside Unity.

Unfortunately I am a complete newby in sound generation and just can’t make it work.

My code looks like this atm:

using UnityEngine;
using System;

namespace RosSharp.RosBridgeClient
{
    public class AudioSubscriber : UnitySubscriber<MessageTypes.AudioCommon.AudioData>
    {
        public int buffer = 320;

        public byte channels = 1;

        //  Sampling rate [Hz]
        public int sample_rate = 16000;

        //  Audio format (e.g. S16LE)
        public string sample_format = "S16LE";

        //  Amount of audio data per second [bits/s]
        public uint bitrate = 128;

        //  Audio coding format (e.g. WAVE, MP3)
        public string coding_format = "WAVE";

        private float[] data;

        private bool audioReceived;
        private bool clipStarted;

        AudioSource source;
        AudioClip.PCMReaderCallback callback;
        AudioClip clip;

        protected override void Start()
        {
            audioReceived = false;
            clipStarted = false;
            source = GetComponent<AudioSource>();
            clip = AudioClip.Create("stream", 320, channels, sample_rate, true, callback);
            source.clip = clip;
            base.Start();
        }

        private void Update()
        {
            if(audioReceived && !clipStarted)
            {
                source.Play();
                clipStarted = true;
            }
        }

        protected override void ReceiveMessage(MessageTypes.AudioCommon.AudioData message)
        {
            //float[] samples = new float[message.data.Length / 4];
            //Buffer.BlockCopy(message.data, 0, samples, 0, message.data.Length);
            float[] floatArr = transformByteToFloatInRange(message.data);
            callback = delegate (float[] data)
            {
                data = floatArr;
            };

            //clip.SetData(data, message.data.Length);
            audioReceived = true;
           
        }

        // Trying to convert the byte between 0 and 255 to float between -1 and 1
        float[] transformByteToFloatInRange(byte[] data)
        {
            float[] floatArr = new float[data.Length];

            for(int i = 0; i < data.Length; i++)
            {
                floatArr[i] = (data[i] / 127.5f) - 1;
            }

            return floatArr;
        }
    }
}

Can anyone maybe see what I am doing wrong and what needs to be changed?

Thanks in advance.

2

Hey!

I don’t think your callback is working as you expect. You should probably callback a function of your class rather than setting a callback every time you receive a message, also you should probably explicitly copy your samples to the data buffer.

public class AudioSubscriber : UnitySubscriber<MessageTypes.AudioCommon.AudioData>
{
    // ...
    // Something to store your received samples
    private Queue<float> sampleQueue = new Queue<float>();

    protected override void Start()
    {
        // ...
        // Here provide Create() with a method of your class as callback
        clip = AudioClip.Create("stream", sample_rate, channels, sample_rate, true, OnAudioRead);
        source.clip = clip;
        base.Start();
    }

    protected override void ReceiveMessage(MessageTypes.AudioCommon.AudioData message)
    {
        // Receive, transform and store your data
        float[] receivedSamples = TransformByteToFloatInRange(message.data);
        foreach (float sample in receivedSamples)
            sampleQueue.Enqueue(sample);

        audioReceived = true;
    }

    void OnAudioRead(float[] data)
    {
        // Fill as much data as you can to the buffer
        for (int i = 0; i < data.Length; i++)
            data[i] = (sampleQueue.Count > 0) ? sampleQueue.Dequeue() : 0;
    }

    // ...
}

Then, I recommend you verify that you’re really receiving 8 bits (0-255) audio data. It seems a bit odd to me as audio is generally encoded on at least 16 bits integers (-32768, 32767).

Cheers!

3

Working with audio data, especially in Unity, can be a bit tricky, especially if you’re new to sound generation. From your code, it seems you’re on the right track, but there are a few things you might want to consider.

  • Byte to Float Conversion: In your transformByteToFloatInRange method, you’re converting bytes to floats. However, the byte values in C# range from 0 to 255, so you might want to adjust your conversion method. Instead of (data / 127.5f) - 1, consider using (data - 128) / 128.0f to get values between -1 and 1.
    - PCMReaderCallback: The AudioClip.PCMReaderCallback delegate is supposed to fill the provided data array. In your callback, you’re just assigning a new reference to the data variable, which might not have the desired effect. Instead, you should copy the values from floatArr to data.
    - Audio Format: Ensure that the audio data you’re receiving is indeed in the WAVE format and matches the sample rate, bit rate, and other parameters you’ve specified. Any discrepancies here can cause playback issues.
    - Playing the AudioClip: You might want to consider using clip.SetData(floatArr, 0) after you’ve converted the byte array to float array. This will set the audio data for the clip, and then you can play it using the AudioSource.
    [/i][/i]
    If you’re still facing issues after trying these suggestions, it might be beneficial to consult with app creators in Miami. They often have experience with various aspects of Unity development, including audio processing, and can provide valuable insights or even direct assistance with your project.