how can I send a render texture over the network?

[system]

hello.

I have a unity server and a client application, both running completely different scenes. I would like to send a render texture that was rendered on the server to the client over the network and display it there on a surface.

How can I achieve this?

i want to send video, but for the moment video compression is not required, ie it’s enough to send single frames at a rate of 5-10 fps. client and server are for this prototype running on the same computer anyway, so transfer rate is not an issue.

[syclamoth]

Sending streaming video data over a network connection isn't really somthing Unity is optimised for. It doesn't provide any serialization classes for video streams, so you'd have to use some kind of 3rd-party plugin for that.

However, for this specific case, is there any reason why you need to run the server and client in completely different scenes? If a client has all the relevant information to recreate a scene on the server, you can just use a camera/render-texture pair to create the picture on clientside, using the (much thinner) position/rotation information of every dynamic object! You'd get better picture quality that way, too.

EDIT: Ok, I'll give you some example code to get you started. I still have my misgivings about this whole idea, but if you're sure you want to do it this way, this is how you'd do it. C# because this is kind of complicated and I don't know enough JS to do it in that.

First up, the texture data from the RenderTexture-

public Color[] GetRenderTexturePixels(RenderTexture tex)
{
    RenderTexture.active = tex;
    Texture2D tempTex = new Texture2D(tex.width, tex.height);
    tempTex.ReadPixels(new Rect(0, 0, tex.width, tex.height), 0, 0);
    tempTex.Apply();
    return tempTex.GetPixels();
}

Now that you have an array of Colors, you can convert this into a byte array for sending it over the network. This is a two-stage process, to account for discrepancies in assembly versions between the standalone players on different architectures.

You'll need to include all of these libraries (phew!).

using System;
using System.IO;
using System.Runtime.Serialization;
using System.Runtime.Serialization.Formatters.Binary;
using System.Reflection;

byte[] SerializeObject<T>(T objectToSerialize)
{
    BinaryFormatter bf = new BinaryFormatter();
    MemoryStream memStr = new MemoryStream();

    bf.Serialize(memStr, objectToSerialize);
memStr.Position = 0;

//return "";
    return memStr.ToArray();
}

T DeserializeObject<T>(byte[] dataStream)
{
    MemoryStream stream = new MemoryStream(dataStream);
    stream.Position = 0;
    BinaryFormatter bf = new BinaryFormatter();
    bf.Binder = new VersionFixer();
    T retV = (T)bf.Deserialize(stream);
    return retV;
}

sealed class VersionFixer : SerializationBinder 
{
    public override Type BindToType(string assemblyName, string typeName) 
    {
        Type typeToDeserialize = null;

        // For each assemblyName/typeName that you want to deserialize to
        // a different type, set typeToDeserialize to the desired type.
        String assemVer1 = Assembly.GetExecutingAssembly().FullName;

        if (assemblyName != assemVer1) 
        {
            // To use a type from a different assembly version, 
            // change the version number.
            // To do this, uncomment the following line of code.
             assemblyName = assemVer1;

            // To use a different type from the same assembly, 
            // change the type name.
        }

        // The following line of code returns the type.
        typeToDeserialize = Type.GetType(String.Format("{0}, {1}", typeName, assemblyName));

        return typeToDeserialize;
    }

}

You need all of that code to serialize the objects, but when that's done you can use

byte[] colourArray = SerializeObject<Color[]>(GetRenderTexturePixels(myRenderTexture));

to convert a rendertexture into a byte array that you can send over the network.

When you have that, just use either an RPC or normal serialization to send the object every network update. I warn you, it's big.

When it arrives at the other end (however you do that), use

Color[] receivedColours = DeserializeObject<Color[]>(receivedBytes);
Texture2D receivedTex = new Texture2D(get the dimensions somehow (I guess 2 extra numbers aren't such a big deal));
receivedTex.SetPixels(receivedColours);
receivedTex.Apply();

Now, you can't shunt that back into a RenderTexture- however, there's not really any need to, since you can just display than on an object now.

EDIT Looks like there's another step in here, too, if you need to use Color32s. We'll have to convert the Color32 array to a type that can be serialized properly, before we can send it over the network. Use something like this-

[System.Serializable]
public class SerializableColor32{
    byte r;
    byte g;
    byte b;
    byte a;
    public SerializableColor32(Color32 source)
    {
        r = source.r;
        g = source.g;
        b = source.b;
        a = source.a;
    }
    public Color32 ConvertToColor32()
    {
        return new Color32(r, g, b, a);
    }
}

public SerializableColor32[] MakeSerializable(Color32[] input)
{
    SerializableColor32[] retV = new SerializableColor32[input.Length];
    for(int i = 0; i < input.Length; i++)
    {
        retV _= new SerializableColor32(input*);*_
 _*}*_
 _*return retV;*_
_*}*_
_*public Color32[] BackToColors(SerializableColor32[] input)*_
_*{*_
 _*Color32[] retV = new Color32[input.Length];*_
 _*for(int i = 0; i < input.Length; i++)*_
 _*{*_
 <em><em>retV _= input*.ConvertToColor32();*_</em></em>
 <em><em>_*}*_</em></em>
 <em><em>_*return retV;*_</em></em>
<em><em>_*}*_</em></em>
<em><em>_*```*_</em></em>
<em><em>_*<p>Then, using those functions, you can do this-</p>*_</em></em>
<em><em>_*```*_</em></em>
<em><em>_*byte[] colourArray = SerializeObject<SerializableColor32[]>(MakeSerializable(GetRenderTexturePixels(myRenderTexture)));*_</em></em>
<em><em>_*```*_</em></em>
<em><em>_*<p>And this-</p>*_</em></em>
<em><em>_*```*_</em></em>
<em><em>_*Color32[] receivedColours = BackToColors(DeserializeObject<SerializableColor32[]>(receivedBytes));*_</em></em>
<em><em>_*```*_</em></em>
<em><em>_*<p><em>(phew)</em></p>*_</em></em>