Suggested way of receiving audio data through TCP conection and update audio clip in Unity

[YujingLiu]

I’m working on a project that receives audio data through TCP communication and then plays it once received.

On the receiving side. I opened another clientReceiveThread for the TCP socket communication, And the receiving code is

    public void ListenForData()
    {
        float[] float_msg_temp;
        float[] float_ms_z;
        float[] float_try = { 3.0f, 3.5F, 4.0F, 4.5F, 5.0F };
        socketConnection = new TcpClient("xxx.xxx.xx.xx", 8010);
        NetworkStream myNetworkStream = socketConnection.GetStream();

        while (true)
        {          
            if (myNetworkStream.CanRead)
            {
                do
                {
                    byte[] myReadBuffer = new byte[1024];
                    float data_len = myNetworkStream.Read(myReadBuffer, 0, myReadBuffer.Length);
                    StringBuilder myCompleteMessage = new StringBuilder();
                    int numberOfBytesRead = 0;
                    // use to change this to float array used by audio clip
                    float_msg_temp = ConvertByteArrayToFloat(myReadBuffer);
                    if (float_msg != null)
                    {
                        float_ms_z = new float[float_msg.Length + float_msg_temp.Length];
                        float_msg.CopyTo(float_ms_z, 0);
                        float_msg_temp.CopyTo(float_ms_z, float_msg.Length);
                        float_msg = float_ms_z;
                    }

                    else
                    {
                        float_ms_z = float_msg_temp;
                        float_msg = float_ms_z;
                    }
                    if (!myNetworkStream.DataAvailable)
                        System.Threading.Thread.Sleep(1);

                } while (myNetworkStream.DataAvailable);
               
                socketConnection.Close();
                CanPlay = true;
                // original idea: play the clip
                Playonce(float_msg);
                socketConnection = new TcpClient("xxx.xxx.xx.xx", 8010);
                myNetworkStream = socketConnection.GetStream();
                Debug.Log("reconnnect ");

            }
            else
            {              
                Debug.Log("Sorry.  You cannot read from this NetworkStream.");
            }
        }
    }

static float[] ConvertByteArrayToFloat(byte[] bytes)
{
    if (bytes.Length % 4 != 0) throw new ArgumentException();

    float[] floats = new float[bytes.Length / 4];
    for (int i = 0; i < floats.Length; i++)
    {
        floats[i] = BitConverter.ToSingle(bytes, i * 4);
    }

    return floats;
}

My original idea was to call a function to play the audio clip

void Playonce(float[] float_msg)

{
    Debug.Log("Play Once");
    if(CanPlay){
        clip = AudioClip.Create("ClipName", float_msg.Length, channels, sampleRate, false);
        clip.SetData(float_msg, 0);
        audioSource.clip = clip;
        audioSource.Play();
        float_msg = null;
        whole_msg = "";
        CanPlay = false;
    }

}

However, I got this error during testing:
UnityException: Construct_Internal can only be called from the main thread.
Constructors and field initializers will be executed from the loading thread when loading a scene.
Don’t use this function in the constructor or field initializers, instead move initialization code to the Awake or Start function.

So I tried another way using the Update function

    void Update()
    {
        if (float_msg != null && CanPlay)
        {
   
            clip = AudioClip.Create("ClipName", float_msg.Length, channels, sampleRate, false);
            clip.SetData(float_msg, 0);
            audioSource.clip = clip;
            audioSource.Play();
            float_msg = null;
            whole_msg = "";
            CanPlay = false;
        }
    }

However, this method just doesn’t feel right to me. And sometimes I have problem of receiving only a short sentence of audio.

I would like to ask

• Is my current TCP connection method a good way of receiving data?
• Is there any better way of updating and playing the audio clip?

[Kurt-Dekker]

@YujingLiu

this method just doesn’t feel right to me.

Any reason why? You can’t do anything with Unity off the main thread, so it seems reasonable to me.

Here’s my blurb about the typical marshal-to-main solution:

Delegate queue to marshal tasks from other threads to the main thread:

How to execute something outside of Update everytime it is called?

You need a delegate queue, then the other thread locks and adds to that queue, while Update locks and removes from it. Then you would package up that little snippet of code above and pass it out for execution on the main thread via Update(): Something as simple as: private Queue<System.Action> TODO = new Queue<System.Action>(); and then to fill it: lock(TODO) { TODO.Enqueue( () => { // your code here playerObj.transform.positio…

@YujingLiu

And sometimes I have problem of receiving only a short sentence of audio.

To debug anything related to Networking, UnityWebRequest, WWW, Postman, curl, WebAPI, etc:

Using Unity for subscription lists

As with all network code, get it working with Postman or curl first. That makes it NOT a Unity problem. Solve the networking first. You must solve that or anything else is irrelevant. Once you have that working, then try and port it to UnityWebRequest. If it still doesn’t work, one handy debug solution is to hook up a proxy like Charles and compare the functioning output from Postman / curl to the output from Unity, and from that reason about where your issues are.

Unity web request acting stupid.

Always get network stuff working outside of Unity in either curl or PostMan, then connect a proxy such as Charles and start trying to replicate the traffic shape with UnityWebRequest. Any other way risks wasting a lot of your time for trivial issues that only the above way will instantly reveal.

And setting up a proxy can be very helpful too, in order to compare traffic:

Using Charles Proxy with Unity