A Standardized Approach to Serialization (With Solution)

UPDATE: Solution here: A Standardized Approach to Serialization (With Solution) - Unity Engine - Unity Discussions


I’m working on writing a standardized approach to saving game data in my projects. I wanted to lay out my goals here, to see if any of you already have favorite methods. Please hit me with whatever you’ve got:

My goals:

  1. During runtime, one Component should keep track of all changes made to its GameObject. When it comes time to create a saved game, all Components of that type are found, and a Save() method is called on each. The opposite should be done with a Load() method during deserialization.

  2. This process should be as invisible as possible. The component should automatically (or with a minimal API) detect when changes are made to any fields (on the GameObject or any of its components), and keep track of what needs to be saved.

  3. Minimal duplication: no field should have its value serialized more than once. If/When this isn’t possible, redundant field serialization should be kept to a minimum.


The main challenge here is detecting changes to field of built-in Unity Components. It’d be great if Unity could automatically detect changes, and mark fields/properties as dirty (with INotifyPropertyChanged, or the like). But in the absence of that feature, I’m working on a few possible ideas.

Do you folks have any ideas? Do any of you use a standard approach to serializing changes in your own games? Many thanks for any thoughts.

-Orion

Why detect changes if you only need to probe the values when Save() is invoked?

By ‘probe the values’, do you mean check each value on Save() to see if it’s changed from some recorded default? Or are you saying you don’t worry about only saving changes, and save all fields that might be changed regardless if they actually were? Or something else? -Thanks.

Since you will most likely write all your data in a single file, why bother comparing values? Take everything and overwrite the existing file with all the current value. Why the comparison?

Just for clarification, I’m not doing any comparisons- that was just part of my question.

I am creating a new save file every time- I think something may have gotten lost in translation here. Let me take another stab:

I don’t want to save the value of every field in my game, regardless of if it’s actually been altered. That could lead to bloated save files that take longer to save and load. Ideally, you’d keep track of what’s changed while playing the game, and only save/load those values. The save file size will grow over time, but it’s only ever as big as it needs to be.

Is this different from how others are doing it? Maybe there’s a standard approach I’m not familiar with.

Alright, I understand you now. You’ll need a initial value and a current value for any variable you want to test/save. When an object is loaded, you will need a script that pick every initial value you want to test against and keep them.

So yeah, it’s a good idea… but you will to assume that every value that can change, could or will change. You want to avoid bloated file size, but if someone play a lot? It’s a bit the issue with Skyrim where the file size just keep increasing as you play.

It’s not necessary to compare values to defaults to see if they’ve changed. All you really need is the information that they have changed (which can be recorded when the value is set, for instance). This approach is used by the Unity editor and old GUI, too (though I bet you already know from your editor script work).

As for the file size increase- that’s true it does grow, but it can’t be any smaller and accurately reflect what’s changed in your game. I suppose maybe with some compression…

Unity does not follow the INotifyPropertyChanged pattern. Their inspector manually flag the object dirty when an Editor GUI is modified. I’m not even sure if Unity performs any kind of check to see if the value changed or not… Probably not, considering how the editor keeps asking to save the scene even if no modification was done.

For some reason, Unity have very little event implemented. Never understood why. Even the new GUI system has 0 proper .NET event.

Unity setting fields/objects as dirty is what I was referring to.

Hey, I’m very interested to hear how far you’ve gotten so far. From what I can see, the most basic underlying construction has to be something like

[Serializable]
public class Wrapper<T> : ScriptableObject
{
    public T val
    {
        get { return _value; }
        set { _value = value;
            changed = true;
        }
    }
    [SerializeField]
    private T _value;
    private bool changed;
}

. I don’t think there is a way to find out whether any field in a MonoBehaviour changed, because that would, like LightStriker said, imply keeping a “previous value” variable for each field, or to deserialize your old data before deciding to serialize your (potentially) new stuff, which could be done via Reflection, but it would either increase memory consumption (plus you’d need to allocate additional memory for lists) or decrease your serialization performance, especially if you want to drill down.
What you are thinking of sounds like some kind of bit manipulation, like, in a bool you could set the second bit depending on whether the variable changed or not (not taking into account here that checks like those on a bool would be much more expensive than to just serialize it). In a class you could probably store that information in the reference, but in a struct you have no such choice. You’d have to inherit from a base struct that contains a bool, and that’s a whole byte more per struct, no matter what you’re doing.
In my streaming engine I therefore only sparingly check for whether blocks of stuff has changed and then serialize those blocks if they have. I deserialize anything, because Unity deserializes anything as well, and if you’re making a custom deserializer for your most important classes, you can trump Unity at speed very easily.

Having said all of this (which probably didn’t turn out to be much use but I needed to write it to get into the topic myself), I’m still very interested to see how you work this out. Have you had a look at this yet ?
http://updatecontrols.codeplex.com/

I think whoever wrote that probably spent a lot of time doing so, and I doubt there will be a better way. But it seems to be the same core problem, though tbh I haven’t had a look inside yet.

Awesome info! Thanks for sharing your approach. Update Controls are very cool, but unfortunately they probably wouldn’t work here, since we need to serialize changes to built-in Unity Components. Many of those values are often fields, not properties.


Ok, I developed a solution that I’m pretty happy with: the Recorder class. It’s sort of like a Facade pattern that also keeps track of what’s changed. It can automatically save and load all changes made to any Component on a GameObject, and will automatically recreate instances of prefabs to load their data as well. It has zero waste (only saves data that’s changed), and when used correctly, zero redundancy (any piece of data is only saved once). It can also correctly save and load Object references of a certain type.

But before getting into the class itself, here’s what it looks like implemented in a derived class:

public class Child : Recorder {

    enum Behaviors { GiveToy }
    private Toy toy; // Toy derives from IDObject

    protected override void OnRecorderAwake () {}

    // Give Toy Behavior
    public void GiveToy(Toy toy) {
        this.toy = toy;
        SetDirty(Behaviors.GiveToy);
    }

    [Serializer(Behaviors.GiveToy)]
    void SaveGiveToy() {
        SaveValue("toy.ID", toy.ID);
    }

    [Deserializer(Behaviors.GiveToy)]
    void LoadGiveToy() {
        string toyID;
        if(TryLoadValue<string>("toy.ID", out toyID)) {
            AddAssignableIDObjectReference(toyID, o => { toy = (Toy)o; });
        }
    }
}

So, you can probably guess a lot from that code, but here’s what’s going on: Everything that can be done to a Child Object is abstracted into high level “behaviors” (in this case, you can “GiveToy” to the Child). These behaviors are defined in an enum, to be used as keys.

Elsewhere in the class, these Behaviors are further defined through methods: “GiveToy()”, for example. This sets an object reference, then calls a “SetDirty()” method with the correct enum value as a key.

immediately below, “Serializer” and “Deserializer” methods are defined. These describe exactly how to save or load everything that could have changed as a result of calling GiveToy(). They are keyed to the same behavior enum using “Serializer” and “Deserializer” attributes. During Awake, these methods will be collected by the Recorder using reflection.

When it comes time to save or load this Child’s data, these methods will be called to do the job. These methods can save and load changes to anything, not just data that’s local to the Recorder. This method can be used to save Transform information, or data from built-in Unity Components.

The Recorder Class:

public abstract class Recorder : IDObject {

    // collection of dirty flags corresponding to actions in the child class (so naughty!)
    private HashSet<int> dirtyBehaviors = new HashSet<int>();

    // methods used to serialize data for each behavior in the child class (keyed by dirty flag)
    private Dictionary<int, Action> serializers = new Dictionary<int, Action>();

    // methods used to deserialize data for each behavior in the child class (keyed by dirty flag)
    private Dictionary<int, Action> deserializers = new Dictionary<int, Action>();

    // used during serialization - set of strings already used as SerializationInfo keys
    private HashSet<string> usedIDs;

    // the SerializationInfo currently being used to save or load
    private SerializationInfo info;

    // have any dirty flags been set by this Recorder
    public bool IsDirty {
        get {
            return dirtyBehaviors.Count() > 0;
        }
    }

    // an serializable object that can be use to save and load this Recorder's data.
    public Data SaveData {
        get {
            return new Data(ID, GetPrefabPath(), IsSceneIDObject, dirtyBehaviors, SaveKeyedData);
        }
    }

    // Awake-analog function called by IDObject
    protected override void OnIDObjectAwake() {

        // collect all Serializer and Deserializer methods in the child class
        GatherSerializationMethods();

        // call Awake-analog method on child
        OnRecorderAwake();
    }

    // pass-through, Awake-analog method on child
    protected abstract void OnRecorderAwake();

    // collects all methods in the child class with a Serializer or Deserializer attribute. Stores these delegates for later use if serializing or deserializing.
    void GatherSerializationMethods() {

        // gather serializers
        IEnumerable<MethodInfo> serializerInfos = GetType().GetMethods(BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Instance).Where(o => o.IsDefined(typeof(Serializer), false));

        foreach(MethodInfo serializerInfo in serializerInfos) {

            Serializer serializer = (Serializer)serializerInfo.GetCustomAttributes(typeof(Serializer), false)[0];

            Action action;
            if(!serializers.TryGetValue(serializer.key, out action)) {

                action = (Action)Delegate.CreateDelegate(typeof(Action), this, serializerInfo.Name);
                serializers.Add(serializer.key, action);
            }
        }

        // gather deserializers
        IEnumerable<MethodInfo> deserializerInfos = GetType().GetMethods(BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Instance).Where(o => o.IsDefined(typeof(Deserializer), false));

        foreach(MethodInfo deserializerInfo in deserializerInfos) {

            Deserializer deserializer = (Deserializer)deserializerInfo.GetCustomAttributes(typeof(Deserializer), false)[0];

            Action action;
            if(!deserializers.TryGetValue(deserializer.key, out action)) {

                action = (Action)Delegate.CreateDelegate(typeof(Action), this, deserializerInfo.Name);
                deserializers.Add(deserializer.key, action);
            }
        }
    }

    // sets a flag corresponding to a behavior in the child class as dirty or clean
    protected void SetDirty(object key, bool dirty = true) {

        if(dirty) {

            dirtyBehaviors.Add((int)key);
        }
        else {

            dirtyBehaviors.Remove((int)key);
        }
    }

    // saves all data for this Recorder
    private void SaveKeyedData(SerializationInfo info) {

        // set info
        this.info = info;

        // set usedKeys
        usedIDs = new HashSet<string>();

        // save keyed data
        foreach(int key in dirtyBehaviors) {

            serializers[key]();
        }

        // remove info
        this.info = null;

        // clear usedKeys
        usedIDs = null;
    }

    // loads all data for this Recorder
    private void LoadKeyedData(SerializationInfo info) {

        // set info
        this.info = info;

        // set usedKeys
        usedIDs = new HashSet<string>();

        // load keyed data
        foreach(int key in dirtyBehaviors) {

            deserializers[key]();
        }

        // remove info
        this.info = null;

        // clear usedKeys
        usedIDs = null;
    }

    // replaces SerializationInfo.AddValue() for use in Recorder serializer methods. Prevents duplicate SerializationInfo keys from being used. If serialized...
    // ...fields are keyed using a string version of their field path, this will prevent duplicate saving of data. This definitely requires some diligence by the...
    // ...author of the child class.
    protected void SaveValue(string id, object value) {

        if(!usedIDs.Contains(id)) {

            info.AddValue(id, value);

            usedIDs.Add(id);
        }
    }

    // replaces SerializationInfo.GetValue<T>() for use in Recorder deserialization methods.
    protected bool TryLoadValue<T>(string id, out T value) {

        if(!usedIDs.Contains(id)) {

            value = (T)info.GetValue(id, typeof(T));

            usedIDs.Add(id);

            return true;
        }
        else {

            value = default(T);

            return false;
        }
    }

    // when deserialized, a Recorder.Data will automatically reinstantiate an instance of the Recorder's prefab if it was originally instantiated during...
    // ...runtime. It calls this method to get the correct prefab path. By default, the IDObject's prefabPath is used, but this can be overridden if needed.
    protected virtual string GetPrefabPath() {

        return PrefabPath;
    }

    // used by the Recorder.Data to find this Recorder's loading method
    public Action<SerializationInfo> GetLoadAction() {

        return LoadKeyedData;
    }

    // use to define serialization methods in the child class.
    [AttributeUsage(AttributeTargets.Method)]
    protected sealed class Serializer : Attribute {

        public int key;

        public Serializer(object key) {

            this.key = (int)key;
        }
    }

    // use to define deserialization methods in the child class.
    [AttributeUsage(AttributeTargets.Method)]
    protected sealed class Deserializer : Attribute {

        public int key;

        public Deserializer(object key) {

            this.key = (int)key;
        }
    }

    // A serializable object that can automatically save and load all needed data for this Recorder. This class should never been instantiated directly...
    // ...Always call Recorder.SaveData to save a specific Recorder's data.
    [Serializable]
    public sealed class Data : ISerializable {

        private string id, prefabPath = string.Empty;
        private bool isSceneObject;
        private HashSet<int> dirtyKeys;
        private Action<SerializationInfo> save;

        public Data() {}

        public Data(string id, string prefabPath, bool isSceneObject, HashSet<int> dirtyKeys, Action<SerializationInfo> save) {

            this.id = id;
            this.prefabPath = prefabPath;
            this.isSceneObject = isSceneObject;
            this.dirtyKeys = dirtyKeys;
            this.save = save;
        }

        // Load
        public Data(SerializationInfo info, StreamingContext context) {

            // load ID
            string id = info.GetString("id");

            // load prefabPath
            string prefabPath = info.GetString("prefabPath");

            // load isSceneObject
            bool isSceneObject = info.GetBoolean("isSceneObject");

            // load dirtyKeys
            int[] dirtyKeysList = (int[])info.GetValue("dirtyKeysList", typeof(int[]));
            dirtyKeys = new HashSet<int>(dirtyKeysList);

            //if ID isn't blank, find Recorder with ID
            if(isSceneObject) {
                Recorder[] recorders = FindObjectsOfType<Recorder>();
                Recorder recorder = Array.Find<Recorder>(recorders, o => o.ID == id);

                if(recorder != null) {

                    recorder.dirtyBehaviors = dirtyKeys;

                    Action<SerializationInfo> load = recorder.GetLoadAction();
                    load(info);
                }
            }

            // else, create a new instance of the prefab
            else if(prefabPath != string.Empty) {

                GameObject prefab = Resources.Load<GameObject>(prefabPath);
                GameObject go = (GameObject)Instantiate(prefab);
                Recorder recorder = go.GetComponent<Recorder>();

                recorder.ID = id;
                IDObject.AssignIDObjectReferences(recorder);

                recorder.dirtyBehaviors = dirtyKeys;
                recorder.LoadKeyedData(info);
            }
        }

        // Save
        void ISerializable.GetObjectData(SerializationInfo info, StreamingContext context) {

            // save ID
            info.AddValue("id", id);

            // save prefabPath
            info.AddValue("prefabPath", prefabPath);

            // save isSceneObject
            info.AddValue("isSceneObject", isSceneObject);

            // save dirtyKeys
            int[] dirtyKeysList = dirtyKeys.ToArray();
            info.AddValue("dirtyKeysList", dirtyKeysList);

            // save recorder
            save(info);
        }
    }
}

There’s a lot of info here, but the Recorder class does a few basic things:

  1. On Awake(), it finds all Serializer and Deserializer methods and stores them as delegates, keyed to their respective, user-defined behaviors.

  2. SetDirty() lets you set a particular behavior as dirty, which just adds an int to a Set of other dirty ints. You can also clean behaviors individually.

  3. Recorder.SavaData is a serializable object. When serialized, it will automatically callback into the Recorder, which will then call the Serializer methods for each of its dirty behaviours.

  4. When deserialized, a Recorder.SavaData will find its original Recorder in your scene and load info about which of its behaviors are dirty. It will then call the Deserializer method for each of its dirty behaviors, loading back all the data.

  5. If the Recorder.SavaData you’re attempting to deserialize was part of a dynamic prefab (instantiated during runtime), it will automatically reinstantiate it and proceed to load its data.

IDObject:

Recorder derives from IDObject. This base object just holds a unique ID and a path to a prefab in a Resources folder (if applicable). It also has some functionality to help save and load references to other IDObjects. Here it is:

public abstract class IDObject : MonoBehaviour {

    // used to reassign object references when loading a save file
    private static Dictionary<string, Action<IDObject>> referencesByID;

    // universally unique ID
    [SerializeField][HideInInspector]
    private string id = string.Empty;
    public virtual string ID {
        get { return id; }
        set { id = value; }
    }

    // path of owning scene, if applicable (relative to project directory)
    [SerializeField][HideInInspector]
    private string scene;
    public virtual string Scene {
        get { return scene; }
        set { scene = value; }
    }

    // is this IDObject saved as part of a scene? True for normal scene objects, prefab instances added to a scene. False for prefabs living in the AssetDatabase.
    [SerializeField][HideInInspector]
    private bool isSceneIDObject = false;
    public bool IsSceneIDObject {
        get { return isSceneIDObject; }
        set { isSceneIDObject = value; }
    }

    // path of base prefab object (relative to Resources folder). Empty if IDObject is not part of a prefab, or if the prefab is not in a Resources folder.
    [SerializeField][HideInInspector]
    private string prefabPath;
    public string PrefabPath {
        get { return prefabPath; }
        set { prefabPath = value; }
    }

    void Awake() {

        // give prefab instances a unique ID
        if(!IsSceneIDObject) {
            ID = UniqueID();
            Scene = string.Empty;
        }

        // call Awake-analog method on child
        OnIDObjectAwake();
    }

    // pass-through, Awake-analog method on child
    protected abstract void OnIDObjectAwake();

    // return a unique ID
    public static string UniqueID() {

        return Guid.NewGuid().ToString();
    }

    // when saving a reference to an IDObject, serialize the string ID instead. When loading, deserialize the string ID, and pass it to this method with a...
    // ...lambda expression assigning the IDObject reference variable. When the IDObject with that ID is later deserialized, it will be passed into your...
    // ...lambda, which will reestablish the object reference.
    public static void AddAssignableIDObjectReference(string key, Action<IDObject> assignReference) {

        IDObject target = FindObjectsOfType<IDObject>().Where(o => o.ID == key).ToArray()[0];

        if(target != null) {

            assignReference(target);
            return;
        }

        if(referencesByID == null) {

            referencesByID = new Dictionary<string, Action<IDObject>>();
        }

        Action<IDObject> assignReferenceDelegate;
        if(referencesByID.TryGetValue(key, out assignReferenceDelegate)) {

            assignReferenceDelegate += assignReference;
        }
        else {

            referencesByID.Add(key, assignReference);
        }
    }

    // pass this method an IDObject to reestablish missing object references stored in referencesByID.
    public static void AssignIDObjectReferences(IDObject idObject) {

        Action<IDObject> assignReferenceDelegate;
        if(referencesByID != null &&
           referencesByID.TryGetValue(idObject.ID, out assignReferenceDelegate)) {

            assignReferenceDelegate(idObject);

            referencesByID.Remove(idObject.ID);
        }
    }
}

(It’s worth noting that objects of this type will not have a unique ID by default. I’m assigning that on my end with a handful of editor scripts, which I’ll be happy to post if anyone’s interested.)

Saving and Loading Your Game:

This is pretty cool. If you use this approach across your game, and make all of your changes to GameObjects by calling behavior methods on Recorders, then saving and loading your entire game could look as simple as this:

public void Save(string fileName) {

    // gather all Recorders and save their data
    using (FileStream saveFileStream = File.Create(fileName)) {
        Recorder.Data[] recorderData = FindObjectsOfType<Recorder>().Select(o => o.SaveData).ToArray();
        new BinaryFormatter().Serialize(saveFileStream, recorderData);
    }
}

public void Load(string fileName) {

    // load all Recorders
    using (Stream saveFileStream = File.OpenRead(fileName)) {
        new BinaryFormatter().Deserialize(saveFileStream);
    }
}

In reality though, if you need to load a scene as part of your loading process, it will likely be a bit more complex. Here’s what I’ve been using at home:

public void Save(string filePath) {

    BinaryFormatter formatter = new BinaryFormatter();
    byte[][] streams = new byte[2][] { new byte[] {}, new byte[] {} };

    // save all level information
    using (MemoryStream levelInfoStream = new MemoryStream()) {
        Recorder levelManager = FindObjectOfType<LevelManager>();
        formatter.Serialize(levelInfoStream, levelManager.SaveData);
        streams[0] = levelInfoStream.ToArray();
    }

    // save all other recorder information
    using (MemoryStream recorderInfoStream = new MemoryStream()) {
        Recorder.Data[] recorderData = FindObjectsOfType<Recorder>().Where(o => !(o is LevelManager)).Where(o => o.IsDirty).Select(o => o.SaveData).ToArray();
        formatter.Serialize(recorderInfoStream, recorderData);
        streams[1] = recorderInfoStream.ToArray();
    }

    // save streams array to file
    using (FileStream saveFileStream = File.Create(filePath)) {
        formatter.Serialize(saveFileStream, streams);
    }
}

IEnumerator Load(string filePath) {

    BinaryFormatter formatter = new BinaryFormatter();
    byte[][] streams = new byte[2][] { new byte[] {}, new byte[] {} };

    // load streams
    using (Stream saveFileStream = File.OpenRead(filePath)) {
        streams = (byte[][])formatter.Deserialize(saveFileStream);
    }

    // load all level inforomation
    using (MemoryStream levelInfoStream = new MemoryStream(streams[0])) {
        formatter.Deserialize(levelInfoStream);
    }

    // load all needed levels
    while(Application.isLoadingLevel) {
        yield return null;
    }

    // load all other recorder information
    using (MemoryStream recorderInfoStream = new MemoryStream(streams[1])) {
        formatter.Deserialize(recorderInfoStream);
    }
}

Saving Object References:

So, IDObject includes some helper functions that let you serialize references to IDObjects. These can be either scene objects, or part of prefab instances instantiated at runtime. You’re not really saving a reference to the object, but saving it’s ID. When loading a saved game, IDObject will find that object by its ID and reestablish your reference to it.

There are two such helper methods: AddAssignableIDObjectReference() (which sets up a missing reference to reestablish), and AssignIDObjectReference (which reestablishes those references). When deriving from Recorder, AssignIDObjectReference() is called for you during deserialization. When writing your Serializer and Deserializer methods, make sure to treat any references to IDObjects like this:

[Serializer(Behaviors.GiveToy)]
void SaveGiveToy() {
    SaveValue("toy.ID", toy.ID);
}

[Deserializer(Behaviors.GiveToy)]
void LoadGiveToy() {
    string toyID;
    if(TryLoadValue<string>("toy.ID", out toyID)) {
        AddAssignableIDObjectReference(toyID, o => { toy = (Toy)o; });
    }
}