Best practices for Generic.Dictionary Serialization?

kiriri · December 6, 2013, 2:25pm

Hello,
I’ve run into a bit of a wall with my current project. I’m storing stuff in a Vector2,Node Dictionary, wherein both Vector2 and Node are custom classes. I’ve read a lot of stuff about this in the past month. Solutions other people say work should be :

Make Dictionary non generic by inheritance.
Write a “Custom Dictionary” class
Write the dictionary keys and values to arrays just before serialization starts and reload them right after.
Override Serialization function in Type?
Implement a custom serializer

If possible I would like to see if we can’t get them all to work. If you have any solution to this it would be awesome if you could post it. I know I’m not the only one having these problems and it might be useful to collect working solutions here.

#Solution 1 does not work. Being generic is not the problem here, being unsorted and non-indexed is.

#Solution 2 is noticeably slower than the original, but if you’re not into micro-optimisation yet or if you just want it for a POC, here is the solution LightStriker kindly posted :

"":

using System;
using System.Collections.Generic;

using UnityEngine;

namespace UniTools
{
    /// <summary>
    /// Dictionary that is Unity serializable.
    /// Simply two lists. Missing some common methods.
    /// </summary>
    /// <typeparam name="Key"></typeparam>
    /// <typeparam name="Value"></typeparam>
    [Serializable]
    public class UniDictionary<Key, Value>
    {
        [SerializeField]
        private List<Key> keys = new List<Key>();

        [SerializeField]
        private List<Value> values = new List<Value>();

        public void Add(Key key, Value value)
        {
            if (keys.Contains(key))
                return;

            keys.Add(key);
            values.Add(value);
        }

        public void Remove(Key key)
        {
            if (!keys.Contains(key))
                return;

            int index = keys.IndexOf(key);

            keys.RemoveAt(index);
            values.RemoveAt(index);
        }

        public bool TryGetValue(Key key, out Value value)
        {
            if (keys.Count != values.Count)
            {
                keys.Clear();
                values.Clear();
                value = default(Value);
                return false;
            }

            if (!keys.Contains(key))
            {
                value = default(Value);
                return false;
            }

            int index = keys.IndexOf(key);
            value = values[index];

            return true;
        }

        public void ChangeValue(Key key, Value value)
        {
            if (!keys.Contains(key))
                return;

            int index = keys.IndexOf(key);

            values[index] = value;
        }
    }
}

#Solution 3, Which atm for my Layer class looks like this :

private bool first = true;
public Layer()// base constructor
        {

                    EditorApplication.playmodeStateChanged += Serialize;

        } 


        public void OnEnable()
        {
            
            OnO();
        }

        public void Serialize()
        {
            if (EditorApplication.isPlayingOrWillChangePlaymode  !EditorApplication.isPlaying)
            { 
                CertainSerialize();
            }
            
        }

        public void CertainSerialize()
        {
            Debug.Log("Saving this many nodes : " + nodes.Keys.Count);

            if (nodes.Count == 0)
                return;
            SerializedNodeKeys = nodes.Keys.ToArray();
            SerializedNodeValues = nodes.Values.ToArray();
        }

        public void OnO()
        {
            if (SerializedNodeKeys == null || SerializedNodeValues == null)
                return;
            Debug.Log("Loading this many nodes : " + SerializedNodeValues.Length);
            nodes = new Dictionary<Vector2, Node>();
            for (var i = 0; i < SerializedNodeValues.Length; i++)
            {
                nodes.Add(SerializedNodeKeys[i],SerializedNodeValues[i]);
            }

        }

Now so far it saves when the user presses the Start button in the editor and it loads the dictionary again both after the game starts and when the game stops. This way no data is lost and changes in play mode are being reverted. Now all we’re missing is a way to serialize the dictionaries just before the compiler wipes them after each compilation. This is a bit more tricky. The only working way I found is to add an editor script which atm needs handle each class that contains a dictionary individually. However, I can imagine this also working in a more generic manner, I just haven’t figured out how to get all instances of the dictionary yet. One could modify the dictionary to add itself to a static list on construction, but I haven’t tried this. Anyways, the working bit of code that detects that the compilation just started is this:

    class MyAllPostprocessor : AssetPostprocessor
    {
        static void OnPostprocessAllAssets(
        string[] importedAssets,
        string[] deletedAssets,
        string[] movedAssets,
        string[] movedFromAssetPaths)
        {
            var coms = UnityEngine.Object.FindObjectsOfType<Layer>();
            foreach (Layer com in coms)
            {
                Debug.Log("Compiler starts");
                com.CertainSerialize();   // 
            }
        }
    }

It needs to be in the editor forlder. Once the compiler finishes. All your scripts that run in edit mode will OnEnable(); in which case you need to load your dics again.

As for
#Solution 4
http://w3mentor.com/learn/asp-dot-net-c-sharp/c-streams/example-of-onserializing-and-ondeserialized/
This means one can use [OnSerializing] and [OnDeserializing] on any function and use a StreamingContext parameter in that function to save and load stuff. However, they do not seem to work in mono 2.3 , at least not from MonoBehaviours or other UnityObjects (come to think of it , I haven’t tried it on normal ones yet).

#Solution 5
I’ve spent at least a week dabbling with custom serializers. And in the end I gained nothing. I started with the popular open source json serializer from whydoidoit.com , but while it is super easy to set up, it is far too heavy on the performance if you have lots of circular references like I do. Then I went on and tried to implement other serializers. The main problem here is that my plugin uses references to unityEngine.GameObject s and to other internal classes which one cannot edit. So all those excellent serializers like protobuff cannot be implemented. Others loose references to said types and then again others are simply far to slow.
So I’ll admit that I won’t try this again. It is probably possible to implement new serializers, but it’s far too big a project to start it just for some dictionaries.

So yeah, that’s my little journey through the exiting world of serialization in Unity. Any help would be appreciated

Dustin-Horne · December 6, 2013, 4:39pm

Well, my JSON .NET serializer works with dictionaries as long as the keys are value types and not reference types. One thing that might cause you big headaches though is trying to serialize a GameObject unless you implement a custom serializer. It should work because according to the Unity documentation, GameObject.rigidbody will return null if one doesn’t exist… however I have a user right now that’s saying Unity is throwing an exception that he’s trying to access a rigidbody but none is attached to the game object. I’m not sure yet whether it’s because of the serializer accessing the “rigidbody” property or whether he’s doing something in one of his scripts…

kiriri · December 9, 2013, 11:06am

then does anyone know how to serialize before compilation starts and how to deserialize right after? This would be needed for any kind of serialization I think.

LightStriker1 · December 9, 2013, 4:01pm

Here, we went with “Make your own class!”

Unity doesn’t like non-ordered collection, such as Dictionary, HashSet and so on. Obviously, serializing a collection that is not indexed is a bit trickier, and the people at Unity decided to not bother.

using System;
using System.Collections.Generic;

using UnityEngine;

namespace UniTools
{
    /// <summary>
    /// Dictionary that is Unity serializable.
    /// Simply two lists. Missing some common methods.
    /// </summary>
    /// <typeparam name="Key"></typeparam>
    /// <typeparam name="Value"></typeparam>
    [Serializable]
    public class UniDictionary<Key, Value>
    {
        [SerializeField]
        private List<Key> keys = new List<Key>();

        [SerializeField]
        private List<Value> values = new List<Value>();

        public void Add(Key key, Value value)
        {
            if (keys.Contains(key))
                return;

            keys.Add(key);
            values.Add(value);
        }

        public void Remove(Key key)
        {
            if (!keys.Contains(key))
                return;

            int index = keys.IndexOf(key);

            keys.RemoveAt(index);
            values.RemoveAt(index);
        }

        public bool TryGetValue(Key key, out Value value)
        {
            if (keys.Count != values.Count)
            {
                keys.Clear();
                values.Clear();
                value = default(Value);
                return false;
            }

            if (!keys.Contains(key))
            {
                value = default(Value);
                return false;
            }

            int index = keys.IndexOf(key);
            value = values[index];

            return true;
        }

        public void ChangeValue(Key key, Value value)
        {
            if (!keys.Contains(key))
                return;

            int index = keys.IndexOf(key);

            values[index] = value;
        }
    }
}

The upside is, serialization works fine.
The downside of this approach is that it is not as fast as a real dictionary.
If you have a huge collection and performance is critical, you could always convert this into a real dictionary post-deserialization.

kiriri · December 10, 2013, 12:17pm

Thanks a lot! I’m sure this will help a lot of people who have similar problems. All the other “solutions” I found via google didn’t work.
It’s too bad that HashSets aren’t supported either (same indexing issue I guess), otherwise this custom dictionary could be a viable alternative for me…

However, I finally figured out how to serialize before and deserialize right after the compilation. I’ve update the example above. It now involves an editor script and it is a bit of a bother to manually do this for all the scripts that contain dictionaries, but it’s better than nothing. Now I just need to work on the performance of the serialization (Unity’s serializer is actually much slower than I would’ve imagined.)

zuzzu · February 20, 2014, 7:06pm

If somebody would be interested, that’s my class addressing this issue. I’ve used an internal dictionary for performance. It supports most methods of the original c# dictionary and is enumerable.

Oh and you have to inherit from that class as Unity doesn’t support generic types serialization afaik.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using UnityEngine;

[Serializable]
public class SerializableDictionary<K, V> : IEnumerable<KeyValuePair<K, V>>
{
	private class Enumerator : IEnumerator<KeyValuePair<K, V>>
	{
		SerializableDictionary<K, V> Dictionary;
		int current = -1;

		public Enumerator(SerializableDictionary<K, V> dictionary)
		{
			Dictionary = dictionary;
		}

		public KeyValuePair<K, V> Current
		{
			get
			{
				return Dictionary.GetAt(current);
			}
		}

		public void Dispose()
		{

		}

		object System.Collections.IEnumerator.Current
		{
			get
			{
				return Dictionary.GetAt(current);
			}
		}

		public bool MoveNext()
		{
			++current;

			return current < Dictionary.Count;
		}

		public void Reset()
		{
			current = -1;
		}
	}

	private Dictionary<K, int> Dictionary = new Dictionary<K, int>();

	[SerializeField]
	private List<K> KeysList = new List<K>();

	[SerializeField]
	private List<V> ValuesList = new List<V>();

	[NonSerialized]
	private bool dictionaryRestored = false;

	public V this[K key]
	{
		get {
			if(!dictionaryRestored)
				RestoreDictionary();

			return ValuesList[Dictionary[key]];
		}

		set {
			if(!dictionaryRestored)
				RestoreDictionary();

			int index;
			if(Dictionary.TryGetValue(key, out index))
			{
				ValuesList[index] = value;
			}
			else
			{
				Dictionary[key] = ValuesList.Count;
				KeysList.Add(key);
				ValuesList.Add(value);
			}
		}
	}

	public V Get(K key, V defaultValue)
	{
		if(!dictionaryRestored)
			RestoreDictionary();

		int index;
		if(Dictionary.TryGetValue(key, out index))
			return ValuesList[index];
		else
			return defaultValue;
	}

	public bool TryGetValue(K key, out V value)
	{
		if(!dictionaryRestored)
			RestoreDictionary();

		int index;
		if(Dictionary.TryGetValue(key, out index))
		{
			value = ValuesList[index];
			return true;
		}
		else
		{
			value = default(V);
			return false;
		}
	}

	public bool Remove(K key)
	{
		if(!dictionaryRestored)
			RestoreDictionary();

		int index;
		if(Dictionary.TryGetValue(key, out index))
		{
			Dictionary.Remove(key);
			KeysList.RemoveAt(index);
			ValuesList.RemoveAt(index);

			int numKeys = KeysList.Count;

			for(int k = index; k < numKeys; ++k)
				--Dictionary[KeysList[k]];

			return true;
		}

		return false;
	}

	public void RemoveAt(int index)
	{
		if(!dictionaryRestored)
			RestoreDictionary();

		K key = KeysList[index];

		Dictionary.Remove(key);
		KeysList.RemoveAt(index);
		ValuesList.RemoveAt(index);
	}

	public KeyValuePair<K, V> GetAt(int index)
	{
		return new KeyValuePair<K, V>(KeysList[index], ValuesList[index]);
	}

	public V GetValueAt(int index)
	{
		return ValuesList[index];
	}

	public int Count
	{
		get {
			return ValuesList.Count;
		}
	}

	public bool ContainsKey(K key)
	{
		if(!dictionaryRestored)
			RestoreDictionary();

		return Dictionary.ContainsKey(key);
	}

	public void Clear()
	{
		Dictionary.Clear();
		KeysList.Clear();
		ValuesList.Clear();
	}

	private void RestoreDictionary()
	{
		for(int i = 0; i < KeysList.Count; ++i)
		{
			Dictionary[KeysList[i]] = i;
		}

		dictionaryRestored = true;
	}

	public IEnumerator<KeyValuePair<K, V>> GetEnumerator()
	{
		return new Enumerator(this);
	}

	System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator()
	{
		return new Enumerator(this);
	}
}

frosted · May 24, 2014, 8:38pm

I found some dictionary code elsewhere that was extremely similar to the above, and it doesn’t work properly if you remove items from the dictionary. The problem is that addition uses the current count as an index, and remove doesn’t shift the remaining indexes. So you end up with corrupt indices and a broken dictionary.

Now, I haven’t actually tested the above code, so I could have missed something and be mistaken. But I’m pretty sure this code does suffer from the same problem - if you’re potentially removing elements be careful.

zuzzu · May 24, 2014, 9:01pm

You’re right. I’ve fixed that some time ago, but that made removals pretty slow. It is probably better to not use that, if you make a lot of updates.

Anyway, I’ve edited the post above to include that fix.

frosted · May 25, 2014, 5:18pm

forgot to change RemoveAt to do the shift. included the shift in removeat here:

public class SerializableDictionary< K, V > : IEnumerable< KeyValuePair< K, V > > {
    public V this[ K key ] {
        get {
            if( !dictionaryRestored )
                RestoreDictionary();
            return ValuesList[ Dictionary[ key ] ];
        }

        set {
            if( !dictionaryRestored )
                RestoreDictionary();
            int index;
            if( Dictionary.TryGetValue( key, out index ) ) {
                ValuesList[ index ] = value;
            } else {
                Add( key, value  );
            }
        }
    }

    public void Add(K key, V value) {
        Dictionary.Add( key, ValuesList.Count );
        KeysList.Add(key);
        ValuesList.Add(value);
    }

    public int Count { get { return ValuesList.Count; } }

    #region IEnumerable<KeyValuePair<K,V>> Members

    public IEnumerator< KeyValuePair< K, V > > GetEnumerator() { return new Enumerator( this ); }

    IEnumerator IEnumerable.GetEnumerator() { return new Enumerator( this ); }

    #endregion

    public V Get( K key, V default_value ) {
        if( !dictionaryRestored )
            RestoreDictionary();

        int index;
        if( Dictionary.TryGetValue( key, out index ) )
            return ValuesList[ index ];
        else
            return default_value;
    }

    
    public bool TryGetValue( K key, out V value ) {
        if( !dictionaryRestored )
            RestoreDictionary();

        int index;

        if( Dictionary.TryGetValue( key, out index ) ) {
            value = ValuesList[ index ];
            return true;
        } else {
            value = default( V );
            return false;
        }
    }


    public bool Remove( K key ) {
        if( !dictionaryRestored )
            RestoreDictionary();


        int index;
        if( Dictionary.TryGetValue( key, out index ) ) {
            RemoveAt( index  );
            return true;
        }
        return false;
    }


    public void RemoveAt( int index ) {
        if( !dictionaryRestored )
            RestoreDictionary();
        K key = KeysList[ index ];
        Dictionary.Remove( key );
        KeysList.RemoveAt( index );
        ValuesList.RemoveAt( index );

        for (int k = index; k < KeysList.Count; ++k)
            --Dictionary[KeysList[k]];
    }

    public KeyValuePair< K, V > GetAt( int index ) { return new KeyValuePair< K, V >( KeysList[ index ], ValuesList[ index ] ); }

    public V GetValueAt( int index ) { return ValuesList[ index ]; }

    public bool ContainsKey( K key ) {
        if( !dictionaryRestored )
            RestoreDictionary();
        return Dictionary.ContainsKey( key );
    }


    public void Clear() {
        Dictionary.Clear();
        KeysList.Clear();
        ValuesList.Clear();
    }


    private void RestoreDictionary() {
        for( int i = 0 ; i < KeysList.Count ; ++i ) {
            Dictionary[ KeysList[ i ] ] = i;
        }

        dictionaryRestored = true;
    }

    private Dictionary< K, int > Dictionary = new Dictionary< K, int >();

    [SerializeField] private List< K > KeysList = new List< K >();

    [SerializeField] private List< V > ValuesList = new List< V >();

    [NonSerialized] private bool dictionaryRestored = false;

    #region Nested type: Enumerator

    private class Enumerator : IEnumerator< KeyValuePair< K, V > > {
        public Enumerator( SerializableDictionary< K, V > dictionary ) { Dictionary = dictionary; }
        #region IEnumerator<KeyValuePair<K,V>> Members

        public KeyValuePair< K, V > Current { get { return Dictionary.GetAt( current ); } }
        public void Dispose() { }
        object IEnumerator.Current { get { return Dictionary.GetAt( current ); } }
        public bool MoveNext() {
            ++current;
            return current < Dictionary.Count;
        }

        public void Reset() { current = -1; }
        #endregion

        private readonly SerializableDictionary< K, V > Dictionary;
        private int current = -1;
    }

    #endregion
}

haven’t fully tested. so use w/ caution

note: it’s probably better to just reimplement dictionary from the base class. just serialize its backing array. If anyone reading this is feeling ambitious

RamzaB · June 4, 2014, 9:20am

Another way that I tested is to serialize the Dictionary into a file through some code, and then implement a custom inspector to view the contents of the Dictionary.

The full source code can be downloaded from here:

Open “SerializeDictionaryTest” scene and run it twice to see the results.

RamzaB · June 6, 2014, 6:31am

Just knew this yesterday from reading an article on Gamasutra , and I thought I’d like to share it here.

Starting from Unity 4.5, Unity supports ISerializationCallbackReceiver which can be used to serialize/deserialize Dictionary.
The example can be located here:

The way the example does it, is by storing dictionary data into lists before doing serialization, and storing the data back into the dictionary after deserialization.
This means that there is an extra step, which may be or may not be a problem.

Unfortunately for me though, this is a problem, but I hope this can help some other people.

winxalex · January 14, 2015, 5:32pm

You choose to use one DataStructure against other cos its upside in current need and goal. You never state that is solution use of totally different data structure with downside in the scenario in the way that List sequentially serach algo compared to Hash table search algo.

Possible solution could be serializing Dictionary in XML with use of DataContractSerializer and the store inside TextAsset as text and then restore it opposite way

TooManySugar · October 2, 2016, 9:54pm

I’m totally stuck ith this.

I whent with option 2 as I found this serializable dictionary that fits my needs:

I use the dictionary without problems but the issue is that I do not know hot to serialize it.
I’ve found no examples of how to serialize a dictionary even if it is serializable.

What I tried was to serialize and save to a file. But I get an error. Not to duplipost you can see the error in the provided link.

Any help would be greatly appreciated.

ioSoftSmith · March 29, 2018, 4:51pm

Resurrect - I was double checking for others solutions for anything I might have missed. Thought this might help someone.

This seems to work for me just fine, (Using JsonUtility.ToJson)

[Serializable]
public class SDictionary<K, V> : Dictionary<K, V>, ISerializationCallbackReceiver
{

    [SerializeField]
    private List<K> m_Keys;
    [SerializeField]
    private List<V> m_Values;
  
    static SDictionary()
    {
        if(!typeof(K).IsSerializable || !typeof(V).IsSerializable)
            Debug.LogError("SDictionary: Types need to be serializable.");
    }


    public void OnBeforeSerialize()
    {
        m_Keys = Keys.ToList();
        m_Values = Values.ToList();
    }

    public void OnAfterDeserialize()
    {
        this.Clear();
        for (int idx = 0; idx < m_Keys.Count; ++idx)
            this.Add(m_Keys[idx], m_Values[idx]);
        m_Keys = null;
        m_Values = null;
    }
}

TooManySugar · March 29, 2018, 6:14pm

Json.net also crunches unity dictionaries without issues.

Topic		Replies	Views
[Solved]How to serialize Dictionary with Unity Serialization System Questions & Answers legacy-topics	9	230651	May 16, 2023
Custom Script Serialization Unity Engine	9	3035	November 9, 2017
Saving and loading Dictionary<> from a file Unity Engine Scripting , Intermediate , Question , 6-2	17	258	November 23, 2025
Edit dictionary in Unity Inspector Unity Engine Scripting , Mono	1	24827	June 5, 2017
Finally, a serializable dictionary for Unity! (extracted from System.Collections.Generic) Unity Engine Scripting	126	238667	May 4, 2024

Best practices for Generic.Dictionary Serialization?

Related topics