Unity / C# little endians and big endians

Unity Engine
1

I am a bit confused on how to deal with little endians and big endians.
I was recently creating methods for converting things to bytes and then back like so…
Click for code

using System;
using UnityEngine;
using System.Collections.Generic;
using System.Collections.ObjectModel;

namespace GameSync
{
    public static class SyncSerialize
    {
        //No new list to avoid garbage collection, though it is unsafe to use so it is read only. Use regular getbytes for new list.
        static List<Byte> getBytes = new List<Byte>(2);
        static ReadOnlyCollection<Byte> getBytesReadOnly = new ReadOnlyCollection<Byte>(getBytes);
        public static ReadOnlyCollection<Byte> GetBytesReadOnly(UInt16 value) { return GetBytesReadOnly((Int16)value); }
        public static ReadOnlyCollection<Byte> GetBytesReadOnly(Int16 value)
        {
            getBytes.Clear();
            getBytes.Add((byte)value);
            getBytes.Add((byte)(value >> 8));
            return getBytesReadOnly;
        }
        public static List<Byte> GetBytes(UInt16 value) { return GetBytes((Int16)value); }
        public static List<Byte> GetBytes(Int16 value)
        {
            return new List<Byte>(GetBytesReadOnly(value));
        }

        public static UInt16 ToUInt16(List<Byte> bytes, int startIndex) { return (UInt16)ToInt16(bytes, startIndex); }
        public static Int16 ToInt16(List<Byte> bytes, int startIndex)
        {
            return (Int16)((bytes[startIndex + 1] << 8) | (bytes[startIndex] << 0));
        }
    }
}

I cant use the c# BitConverter methods since they cause garbage collection and I plan on using this for networking, meaning it will be ran very frequently.

I later stumbled upon little endians and big endians.
To my understanding, the way I am handling things above is creating a byte list in little endian order.
I have read things like how c# handles everything in little endians and how networking is done in big endians etc…

My question is, do I need to worry about any of this? I plan on using this code to not only send packets over the network using unitys NetworkTransport.Send, but to also save the bytes in a file for game replays.
If someone on a little endians machine played my game and someone on a big endians machine played my game and used my Serialize class, will they not be able to communicate through the network? If the big endian sent a replay file that was created using my Serialize class (saved with something like File.WriteAllBytes(“MyReplay.replay”, bytesArray):wink: and sent it to the little endians computer, will there be problems?
Am I going to have to check BitConverter.IsLittleEndian for everything that has to do with bytes and convert it?
When do I have to worry about this stuff?

I have seen this thread Handling of the bitwise/endian problem? - Unity Engine - Unity Discussions , but it is from 2006 and I don’t know what they mean by “Unity handles it”
Are they still handling it? What are they handling and how?
I am assuming my code will work on any machine as long as the bytes were created in little endians, which they are in my class, regardless to what machine the bytes were created on, and regardless to what network (internet) they use.

If anyone has answers, please let me know.
Thanks in advance =)

2

When you transfer byte data, you’re transferring them as bytes. So the endianness doesn’t matter. The format of your byte stream is dependent completely on the order that you stream the bytes. Endianness does not have to do with the order of the bits in a single byte, but rather the order of the bytes in a word (4 byte words in a 32-bit system, and 8 byte words in a 64-bit system).

Your methods you use to get the bytes decide the endian order regardless of the system. For example your ToInt16 presumes the byte collection coming in has the bytes in little endian order (the first byte at startIndex is the lower half of the 2 byte Int16).

So no, you won’t have an issue in this regard.

3

So even if I use something for strings like so…
(sorry for how messy it seems, its a pain in the ass to avoid garbage collection T.T)
Click for code

        static List<byte> getBytes = new List<byte>();
        static ReadOnlyCollection<byte> getBytesReadOnly = new ReadOnlyCollection<byte>(getBytes);

        public static ReadOnlyCollection<byte> GetBytesReadOnly(string value)
        {
            getBytes.Clear();

            for(int i = 0; i < value.Length; i++)
            {
                getBytes.Add((byte)(value[i]));
            }

            return getBytesReadOnly;
        }

        public static string ToString(List<byte> bytes)
        {
            System.Text.StringBuilder stringBuilder = new System.Text.StringBuilder();

            for(int i = 0; i < bytes.Count; i++)
            {
                stringBuilder.Append((char)bytes[i]); //stringbuilder.append(list) didnt seem to work properly
            }

            return stringBuilder.ToString();
        }

I should still be safe?

In other words, little endians and big endians are basically just encoding types. Some things were created with a specific encoding type, and to read it you need to use that encoding type. But then why are people worried about cross platform problems with their code? If their code was designed to work with a certain type of endian, then there should be no issue.
Or are they just worried about outside content being used with their software? For example, I have a program that can take in text and replace all the P’s with Q’s, and someone typed in a Notepad on a computer that was big endian based, copied the text from there and pasted it into my program, but my program expects little endians.

4

You’re still doing it byte by byte.

Endianess doesn’t effect your byte to byte values, as it has to do with the shape of a word worth of bytes in memory. It’s sort of like encoding, yeah, you could say that. And like I said, it effects bytes grouped together in memory… so if you break things up into bytes yourself, endianess has nothing to do with it anymore.

Read about it here:

What you should be concerned about with your string bit converter is that you treat your char values as 1 byte, when .net’s char is actually a 16-bit UTF-16 value… or 2 bytes per character:

If you feed in a string that contains letters outside of the first 256 values represented, you’ll get errors.

Also, I get you’re trying to avoid all the byte[ ] array objects being created by BitConverter. But you’re just recycling the same List over and over.

You must be sticking those somewhere every time you convert to bytes some value, right?

Is it possibly a ‘Stream’? Because… .Net already has tools to write bytes to streams while avoid the garbage associated with BitConverter. Namely there is ‘BinaryWriter’:

And of course on the receiving end, you can use binary reader:
https://msdn.microsoft.com/en-us/library/system.io.binaryreader(v=vs.110).aspx

Even if you’re not using a stream, and you’re just piling these all into one large bytearray… you can fill the stream and once done get the byte array from that.

byte[] result;

using(var strm as new MemoryStream())
using(var writer as new BinaryWriter(strm))
{
   writer.Write(someInt);
   writer.Write(someFloat);
   writer.Write(someBool);
   result = strm.ToArray();
}

But, really, usually when sending data over a network you have some network stream to work with. And in that regard, you just open the stream, hand it to the BinaryWriter, and start writing your sequence into the stream.

[edit]
Although, I see you said you’re using NetworkTransport.Send, in which case, yeah… use my demo code above.

Which you’d just feed that byte array into when done.

Also note, that usually when transferring data you have to serialize it. And when doing that you usually have some serialization pattern. You could just have a serializable pattern that takes in a stream, writes its values to it, and then send that over.

This is actually how the built in .Net ISerializable pattern works. You create a Formatter (defines how stuff is written, you’d use a BinaryFormatter), and then you create a stream, then feed the objects you need serialized into the formatter:

[System.Serializable()]
public struct DataStructure
{

   public int Id;
   public string Name;
   public float Health;
   //... so on so forth

}



var data = new DataStructure();
data.Id = 5;
data.Name = "Hello World";
data.Health = 99f;

var formatter = new BinaryFormatter();
using (var strm = new MemoryStream())
{
   formatter.Serialize(strm, data);
   byte error;
   NetworkTransport.Send(hostId, connectionId, channelId, strm.ToArray(), strm.Length, out error);
}
1 Like
5

Good catch =). I guess I will have to find a way to encode the string to utf-8 first without using the Encoding.UTF8.GetBytes due to garbage collection or just live with having 2 bytes per char / the garbage…

Since I return a readonly, I have the option to either .AddRange the readonly to another list that I am also handling garbage collection with, or I can just read the values and do what I desire right then and there (not thread safe I am assuming).
I avoid .ToArray due to garbage as well. Basically what I do is, if I have something that runs very frequently, I cant use any method that utilizes arrays and such unless I do what I did above with using a single array and clearing it.

I will have to learn about binaryformatters, streams and such.

The way I was going to do things is have Sync components that can send to a central area information such as the Sync object byte[ ] constant size or dynamic size. This central component will keep a consistent order of the sync components and separate their bytes[ ] depending on the size given. It will Send the bytes if the object is the clients object, and it will receive the appropriate bytes if the object is either the client or not. Then it will send to the Sync Components their appropriate bytes and they each will deserialize as needed.
At the same time, I will be able to record all the bytes, and be able to play things back. So the Sync components are whats serializing and deserializing, but they are not sending anything over the network. That is done at a central location.
So basically I am trying to combine the replay system and the network system (or more accurately, have the network system utilize the replay/record system).

6

do note… lists have arrays inside of them, and garbage will collect if it has to resize the array to fit more elements.

Also, you will need an array at some point to send it through the NetworkTransport since it only takes an array.

Lastly, a stream can be written to an existing array with the ‘Read’ method. So you can create a reusable array that you write the stream to before sending it off. This is actually why NetworkTransport.Send has a ‘length’ value on it. So that you can have a large reusable array that you might only fill up half way, and you tell NetworkTransport what portion of the array to pay attention to.

2 Likes