Finally, a serializable dictionary for Unity! (extracted from System.Collections.Generic)

Hey guys,

so as all you all know Unity doesn't know how to serialize generic dictionaries. Since 4.6 we got the ISerializationCallbackReceiver which allows us to use custom serialization to serialize types that Unity cannot. However; when using this interface to serialize dictionaries it's not idea. For example, if you inherit Dictionary and add the serialized key/values lists, you'll come across some really weird issues (most probably threading related) as seen here and here. Another approach is to use composition instead of inheritance, and let the dictionary object live within the serialized dictionary class alongside the serialized key/value lists, i.e.

public class SerializableDictionary<TK, TV> : ISerializationCallbackReceiver
{
    private Dictionary<TK, TV> _Dictionary;
    [SerializeField] List<TK> _Keys;
    [SerializeField] List<TV> _Values;

    // wrapper methods, serialization, etc...
}

instead of:

public class SerializableDictionary<TK, TV> : Dictionary<TK, TV>, ISerializationCallbackReceiver
{
    [SerializeField] List<TK> _Keys;
    [SerializeField] List<TV> _Values;

    // serialization, etc...
}

But that also feels redundant to me, we're having to write wrappers and add two lists just to serialize...

Wouldn't it be nice if the Dictionary class itself was serializable by Unity?

Well that's what I did by basically extracting the Dictionary code and marking the necessary fields with [SerializeField] - And slightly refactored/cleaned it up (renamed the fields, removed the implementation of the non-generic interfaces, and factored out the 'hashCode', 'next', 'key', and 'value' from the 'Entry' class to be arrays instead)

One bit I left commented was the "IsWellKnownEqualityComparer" as I wasn't really sure about it - I did some tests and it didn't seem to affect anything. If you know for a fact it's critical let me know.

Of course, you'd still have to subclass, i.e.

[Serializable]
public class MyDictionary : SerializableDictionary<string, int> { }

public class Test : MonoBehaviour
{
    public MyDictionary dictionary;
}

I've also added an indexer that accepts a default value to return if the key wasn't found. As well as an AsDictionary property which is mainly for debugging reasons.

using System;
using System.Linq;
using System.Collections;
using System.Collections.Generic;
using System.Diagnostics;
using UnityEngine;

[Serializable, DebuggerDisplay("Count = {Count}")]
public class SerializableDictionary<TKey, TValue> : IDictionary<TKey, TValue>
{
    [SerializeField, HideInInspector] int[] _Buckets;
    [SerializeField, HideInInspector] int[] _HashCodes;
    [SerializeField, HideInInspector] int[] _Next;
    [SerializeField, HideInInspector] int _Count;
    [SerializeField, HideInInspector] int _Version;
    [SerializeField, HideInInspector] int _FreeList;
    [SerializeField, HideInInspector] int _FreeCount;
    [SerializeField, HideInInspector] TKey[] _Keys;
    [SerializeField, HideInInspector] TValue[] _Values;

    readonly IEqualityComparer<TKey> _Comparer;

    // Mainly for debugging purposes - to get the key-value pairs display
    public Dictionary<TKey, TValue> AsDictionary
    {
        get { return new Dictionary<TKey, TValue>(this); }
    }

    public int Count
    {
        get { return _Count - _FreeCount; }
    }

    public TValue this[TKey key, TValue defaultValue]
    {
        get
        {
            int index = FindIndex(key);
            if (index >= 0)
                return _Values[index];
            return defaultValue;
        }
    }

    public TValue this[TKey key]
    {
        get
        {
            int index = FindIndex(key);
            if (index >= 0)
                return _Values[index];
            throw new KeyNotFoundException(key.ToString());
        }

        set { Insert(key, value, false); }
    }

    public SerializableDictionary()
        : this(0, null)
    {
    }

    public SerializableDictionary(int capacity)
        : this(capacity, null)
    {
    }

    public SerializableDictionary(IEqualityComparer<TKey> comparer)
        : this(0, comparer)
    {
    }

    public SerializableDictionary(int capacity, IEqualityComparer<TKey> comparer)
    {
        if (capacity < 0)
            throw new ArgumentOutOfRangeException("capacity");

        Initialize(capacity);

        _Comparer = (comparer ?? EqualityComparer<TKey>.Default);
    }

    public SerializableDictionary(IDictionary<TKey, TValue> dictionary)
        : this(dictionary, null)
    {
    }

    public SerializableDictionary(IDictionary<TKey, TValue> dictionary, IEqualityComparer<TKey> comparer)
        : this((dictionary != null) ? dictionary.Count : 0, comparer)
    {
        if (dictionary == null)
            throw new ArgumentNullException("dictionary");

        foreach (KeyValuePair<TKey, TValue> current in dictionary)
            Add(current.Key, current.Value);
    }

    public bool ContainsValue(TValue value)
    {
        if (value == null)
        {
            for (int i = 0; i < _Count; i++)
            {
                if (_HashCodes[i] >= 0 && _Values[i] == null)
                    return true;
            }
        }
        else
        {
            var defaultComparer = EqualityComparer<TValue>.Default;
            for (int i = 0; i < _Count; i++)
            {
                if (_HashCodes[i] >= 0 && defaultComparer.Equals(_Values[i], value))
                    return true;
            }
        }
        return false;
    }

    public bool ContainsKey(TKey key)
    {
        return FindIndex(key) >= 0;
    }

    public void Clear()
    {
        if (_Count <= 0)
            return;

        for (int i = 0; i < _Buckets.Length; i++)
            _Buckets[i] = -1;

        Array.Clear(_Keys, 0, _Count);
        Array.Clear(_Values, 0, _Count);
        Array.Clear(_HashCodes, 0, _Count);
        Array.Clear(_Next, 0, _Count);

        _FreeList = -1;
        _Count = 0;
        _FreeCount = 0;
        _Version++;
    }

    public void Add(TKey key, TValue value)
    {
        Insert(key, value, true);
    }

    private void Resize(int newSize, bool forceNewHashCodes)
    {
        int[] bucketsCopy = new int[newSize];
        for (int i = 0; i < bucketsCopy.Length; i++)
            bucketsCopy[i] = -1;

        var keysCopy = new TKey[newSize];
        var valuesCopy = new TValue[newSize];
        var hashCodesCopy = new int[newSize];
        var nextCopy = new int[newSize];

        Array.Copy(_Values, 0, valuesCopy, 0, _Count);
        Array.Copy(_Keys, 0, keysCopy, 0, _Count);
        Array.Copy(_HashCodes, 0, hashCodesCopy, 0, _Count);
        Array.Copy(_Next, 0, nextCopy, 0, _Count);

        if (forceNewHashCodes)
        {
            for (int i = 0; i < _Count; i++)
            {
                if (hashCodesCopy[i] != -1)
                    hashCodesCopy[i] = (_Comparer.GetHashCode(keysCopy[i]) & 2147483647);
            }
        }

        for (int i = 0; i < _Count; i++)
        {
            int index = hashCodesCopy[i] % newSize;
            nextCopy[i] = bucketsCopy[index];
            bucketsCopy[index] = i;
        }

        _Buckets = bucketsCopy;
        _Keys = keysCopy;
        _Values = valuesCopy;
        _HashCodes = hashCodesCopy;
        _Next = nextCopy;
    }

    private void Resize()
    {
        Resize(PrimeHelper.ExpandPrime(_Count), false);
    }

    public bool Remove(TKey key)
    {
        if (key == null)
            throw new ArgumentNullException("key");

        int hash = _Comparer.GetHashCode(key) & 2147483647;
        int index = hash % _Buckets.Length;
        int num = -1;
        for (int i = _Buckets[index]; i >= 0; i = _Next[i])
        {
            if (_HashCodes[i] == hash && _Comparer.Equals(_Keys[i], key))
            {
                if (num < 0)
                    _Buckets[index] = _Next[i];
                else
                    _Next[num] = _Next[i];

                _HashCodes[i] = -1;
                _Next[i] = _FreeList;
                _Keys[i] = default(TKey);
                _Values[i] = default(TValue);
                _FreeList = i;
                _FreeCount++;
                _Version++;
                return true;
            }
            num = i;
        }
        return false;
    }

    private void Insert(TKey key, TValue value, bool add)
    {
        if (key == null)
            throw new ArgumentNullException("key");

        if (_Buckets == null)
            Initialize(0);

        int hash = _Comparer.GetHashCode(key) & 2147483647;
        int index = hash % _Buckets.Length;
        int num1 = 0;
        for (int i = _Buckets[index]; i >= 0; i = _Next[i])
        {
            if (_HashCodes[i] == hash && _Comparer.Equals(_Keys[i], key))
            {
                if (add)
                    throw new ArgumentException("Key already exists: " + key);

                _Values[i] = value;
                _Version++;
                return;
            }
            num1++;
        }
        int num2;
        if (_FreeCount > 0)
        {
            num2 = _FreeList;
            _FreeList = _Next[num2];
            _FreeCount--;
        }
        else
        {
            if (_Count == _Keys.Length)
            {
                Resize();
                index = hash % _Buckets.Length;
            }
            num2 = _Count;
            _Count++;
        }
        _HashCodes[num2] = hash;
        _Next[num2] = _Buckets[index];
        _Keys[num2] = key;
        _Values[num2] = value;
        _Buckets[index] = num2;
        _Version++;

        //if (num3 > 100 && HashHelpers.IsWellKnownEqualityComparer(comparer))
        //{
        //    comparer = (IEqualityComparer<TKey>)HashHelpers.GetRandomizedEqualityComparer(comparer);
        //    Resize(entries.Length, true);
        //}
    }

    private void Initialize(int capacity)
    {
        int prime = PrimeHelper.GetPrime(capacity);

        _Buckets = new int[prime];
        for (int i = 0; i < _Buckets.Length; i++)
            _Buckets[i] = -1;

        _Keys = new TKey[prime];
        _Values = new TValue[prime];
        _HashCodes = new int[prime];
        _Next = new int[prime];

        _FreeList = -1;
    }

    private int FindIndex(TKey key)
    {
        if (key == null)
            throw new ArgumentNullException("key");

        if (_Buckets != null)
        {
            int hash = _Comparer.GetHashCode(key) & 2147483647;
            for (int i = _Buckets[hash % _Buckets.Length]; i >= 0; i = _Next[i])
            {
                if (_HashCodes[i] == hash && _Comparer.Equals(_Keys[i], key))
                    return i;
            }
        }
        return -1;
    }

    public bool TryGetValue(TKey key, out TValue value)
    {
        int index = FindIndex(key);
        if (index >= 0)
        {
            value = _Values[index];
            return true;
        }
        value = default(TValue);
        return false;
    }

    private static class PrimeHelper
    {
        public static readonly int[] Primes = new int[]
        {
            3,
            7,
            11,
            17,
            23,
            29,
            37,
            47,
            59,
            71,
            89,
            107,
            131,
            163,
            197,
            239,
            293,
            353,
            431,
            521,
            631,
            761,
            919,
            1103,
            1327,
            1597,
            1931,
            2333,
            2801,
            3371,
            4049,
            4861,
            5839,
            7013,
            8419,
            10103,
            12143,
            14591,
            17519,
            21023,
            25229,
            30293,
            36353,
            43627,
            52361,
            62851,
            75431,
            90523,
            108631,
            130363,
            156437,
            187751,
            225307,
            270371,
            324449,
            389357,
            467237,
            560689,
            672827,
            807403,
            968897,
            1162687,
            1395263,
            1674319,
            2009191,
            2411033,
            2893249,
            3471899,
            4166287,
            4999559,
            5999471,
            7199369
        };

        public static bool IsPrime(int candidate)
        {
            if ((candidate & 1) != 0)
            {
                int num = (int)Math.Sqrt((double)candidate);
                for (int i = 3; i <= num; i += 2)
                {
                    if (candidate % i == 0)
                    {
                        return false;
                    }
                }
                return true;
            }
            return candidate == 2;
        }

        public static int GetPrime(int min)
        {
            if (min < 0)
                throw new ArgumentException("min < 0");

            for (int i = 0; i < PrimeHelper.Primes.Length; i++)
            {
                int prime = PrimeHelper.Primes[i];
                if (prime >= min)
                    return prime;
            }
            for (int i = min | 1; i < 2147483647; i += 2)
            {
                if (PrimeHelper.IsPrime(i) && (i - 1) % 101 != 0)
                    return i;
            }
            return min;
        }

        public static int ExpandPrime(int oldSize)
        {
            int num = 2 * oldSize;
            if (num > 2146435069 && 2146435069 > oldSize)
            {
                return 2146435069;
            }
            return PrimeHelper.GetPrime(num);
        }
    }

   public ICollection<TKey> Keys
   {
        get { return _Keys.Take(Count).ToArray(); }
   }

   public ICollection<TValue> Values
   {
        get { return _Values.Take(Count).ToArray(); }
   }

    public void Add(KeyValuePair<TKey, TValue> item)
    {
        Add(item.Key, item.Value);
    }

    public bool Contains(KeyValuePair<TKey, TValue> item)
    {
        int index = FindIndex(item.Key);
        return index >= 0 &&
            EqualityComparer<TValue>.Default.Equals(_Values[index], item.Value);
    }

    public void CopyTo(KeyValuePair<TKey, TValue>[] array, int index)
    {
        if (array == null)
            throw new ArgumentNullException("array");

        if (index < 0 || index > array.Length)
            throw new ArgumentOutOfRangeException(string.Format("index = {0} array.Length = {1}", index, array.Length));

        if (array.Length - index < Count)
            throw new ArgumentException(string.Format("The number of elements in the dictionary ({0}) is greater than the available space from index to the end of the destination array {1}.", Count, array.Length));

        for (int i = 0; i < _Count; i++)
        {
            if (_HashCodes[i] >= 0)
                array[index++] = new KeyValuePair<TKey, TValue>(_Keys[i], _Values[i]);
        }
    }

    public bool IsReadOnly
    {
        get { return false; }
    }

    public bool Remove(KeyValuePair<TKey, TValue> item)
    {
        return Remove(item.Key);
    }

    public Enumerator GetEnumerator()
    {
        return new Enumerator(this);
    }

    IEnumerator IEnumerable.GetEnumerator()
    {
        return GetEnumerator();
    }

    IEnumerator<KeyValuePair<TKey, TValue>> IEnumerable<KeyValuePair<TKey, TValue>>.GetEnumerator()
    {
        return GetEnumerator();
    }

    public struct Enumerator : IEnumerator<KeyValuePair<TKey, TValue>>
    {
        private readonly SerializableDictionary<TKey, TValue> _Dictionary;
        private int _Version;
        private int _Index;
        private KeyValuePair<TKey, TValue> _Current;

        public KeyValuePair<TKey, TValue> Current
        {
            get { return _Current; }
        }

        internal Enumerator(SerializableDictionary<TKey, TValue> dictionary)
        {
            _Dictionary = dictionary;
            _Version = dictionary._Version;
            _Current = default(KeyValuePair<TKey, TValue>);
            _Index = 0;
        }

        public bool MoveNext()
        {
            if (_Version != _Dictionary._Version)
                throw new InvalidOperationException(string.Format("Enumerator version {0} != Dictionary version {1}", _Version, _Dictionary._Version));

            while (_Index < _Dictionary._Count)
            {
                if (_Dictionary._HashCodes[_Index] >= 0)
                {
                    _Current = new KeyValuePair<TKey, TValue>(_Dictionary._Keys[_Index], _Dictionary._Values[_Index]);
                    _Index++;
                    return true;
                }
                _Index++;
            }

            _Index = _Dictionary._Count + 1;
            _Current = default(KeyValuePair<TKey, TValue>);
            return false;
        }

        void IEnumerator.Reset()
        {
            if (_Version != _Dictionary._Version)
                throw new InvalidOperationException(string.Format("Enumerator version {0} != Dictionary version {1}", _Version, _Dictionary._Version));

            _Index = 0;
            _Current = default(KeyValuePair<TKey, TValue>);
        }

        object IEnumerator.Current
        {
            get { return Current; }
        }

        public void Dispose()
        {
        }
    }
}

A property drawer will come soon!

Cheers!

27 Likes

So here's a drawer, not the best, but hey it works!

  • Does not support reference types as keys (except for strings)
  • You'd have to add a new property drawer for each dictionary type you make (see the bottom of the script)
  • The drawer is open for extension by community, please feel free to improve it!
  • Again, for better dictionary inspection see my framework. I offer per key/value attribute annotation!
[Serializable] public class MyDictionary1 : SerializableDictionary<string, int> { }
[Serializable] public class MyDictionary2 : SerializableDictionary<KeyCode, GameObject> { }

public class Test : MonoBehaviour
{
  public MyDictionary1 dictionary1;
  public MyDictionary2 dictionary2;
}
using System;
using System.Collections.Generic;
using UnityEditor;
using UnityEngine;
using UnityObject = UnityEngine.Object;

public abstract class DictionaryDrawer<TK, TV> : PropertyDrawer
{
    private SerializableDictionary<TK, TV> _Dictionary;
    private bool _Foldout;
    private const float kButtonWidth = 18f;

    public override float GetPropertyHeight(SerializedProperty property, GUIContent label)
    {
        CheckInitialize(property, label);
        if (_Foldout)
            return (_Dictionary.Count + 1) * 17f;
        return 17f;
    }

    public override void OnGUI(Rect position, SerializedProperty property, GUIContent label)
    {
        CheckInitialize(property, label);

        position.height = 17f;

        var foldoutRect = position;
        foldoutRect.width -= 2 * kButtonWidth;
        EditorGUI.BeginChangeCheck();
        _Foldout = EditorGUI.Foldout(foldoutRect, _Foldout, label, true);
        if (EditorGUI.EndChangeCheck())
            EditorPrefs.SetBool(label.text, _Foldout);

        var buttonRect = position;
        buttonRect.x = position.width - kButtonWidth + position.x;
        buttonRect.width = kButtonWidth + 2;

        if (GUI.Button(buttonRect, new GUIContent("+", "Add item"), EditorStyles.miniButton))
        {
            AddNewItem();
        }

        buttonRect.x -= kButtonWidth;

        if (GUI.Button(buttonRect, new GUIContent("X", "Clear dictionary"), EditorStyles.miniButtonRight))
        {
            ClearDictionary();
        }

        if (!_Foldout)
            return;

        foreach (var item in _Dictionary)
        {
            var key = item.Key;
            var value = item.Value;

            position.y += 17f;

            var keyRect = position;
            keyRect.width /= 2;
            keyRect.width -= 4;
            EditorGUI.BeginChangeCheck();
            var newKey = DoField(keyRect, typeof(TK), key);
            if (EditorGUI.EndChangeCheck())
            {
                try
                {
                    _Dictionary.Remove(key);
                    _Dictionary.Add(newKey, value);
                }
                catch(Exception e)
                {
                    Debug.Log(e.Message);
                }
                break;
            }

            var valueRect = position;
            valueRect.x = position.width / 2 + 15;
            valueRect.width = keyRect.width - kButtonWidth;
            EditorGUI.BeginChangeCheck();
            value = DoField(valueRect, typeof(TV), value);
            if (EditorGUI.EndChangeCheck())
            {
                _Dictionary[key] = value;
                break;
            }

            var removeRect = valueRect;
            removeRect.x = valueRect.xMax + 2;
            removeRect.width = kButtonWidth;
            if (GUI.Button(removeRect, new GUIContent("x", "Remove item"), EditorStyles.miniButtonRight))
            {
                RemoveItem(key);
                break;
            }
        }
    }

    private void RemoveItem(TK key)
    {
        _Dictionary.Remove(key);
    }

    private void CheckInitialize(SerializedProperty property, GUIContent label)
    {
        if (_Dictionary == null)
        {
            var target = property.serializedObject.targetObject;
            _Dictionary = fieldInfo.GetValue(target) as SerializableDictionary<TK, TV>;
            if (_Dictionary == null)
            {
                _Dictionary = new SerializableDictionary<TK, TV>();
                fieldInfo.SetValue(target, _Dictionary);
            }

            _Foldout = EditorPrefs.GetBool(label.text);
        }
    }

    private static readonly Dictionary<Type, Func<Rect, object, object>> _Fields =
        new Dictionary<Type,Func<Rect,object,object>>()
        {
            { typeof(int), (rect, value) => EditorGUI.IntField(rect, (int)value) },
            { typeof(float), (rect, value) => EditorGUI.FloatField(rect, (float)value) },
            { typeof(string), (rect, value) => EditorGUI.TextField(rect, (string)value) },
            { typeof(bool), (rect, value) => EditorGUI.Toggle(rect, (bool)value) },
            { typeof(Vector2), (rect, value) => EditorGUI.Vector2Field(rect, GUIContent.none, (Vector2)value) },
            { typeof(Vector3), (rect, value) => EditorGUI.Vector3Field(rect, GUIContent.none, (Vector3)value) },
            { typeof(Bounds), (rect, value) => EditorGUI.BoundsField(rect, (Bounds)value) },
            { typeof(Rect), (rect, value) => EditorGUI.RectField(rect, (Rect)value) },
        };

    private static T DoField<T>(Rect rect, Type type, T value)
    {
        Func<Rect, object, object> field;
        if (_Fields.TryGetValue(type, out field))
            return (T)field(rect, value);

        if (type.IsEnum)
            return (T)(object)EditorGUI.EnumPopup(rect, (Enum)(object)value);

        if (typeof(UnityObject).IsAssignableFrom(type))
            return (T)(object)EditorGUI.ObjectField(rect, (UnityObject)(object)value, type, true);

        Debug.Log("Type is not supported: " + type);
        return value;
    }

    private void ClearDictionary()
    {
        _Dictionary.Clear();
    }

    private void AddNewItem()
    {
        TK key;
        if (typeof(TK) == typeof(string))
            key = (TK)(object)"";
        else key = default(TK);

        var value = default(TV);
        try
        {
            _Dictionary.Add(key, value);
        }
        catch(Exception e)
        {
            Debug.Log(e.Message);
        }
    }
}

[CustomPropertyDrawer(typeof(MyDictionary1))]
public class MyDictionaryDrawer1 : DictionaryDrawer<string, int> { }

[CustomPropertyDrawer(typeof(MyDictionary2))]
public class MyDictionaryDrawer2 : DictionaryDrawer<KeyCode, GameObject> { }
12 Likes

So returning the keys/values directly in the Keys/Values property is not a good idea since their size is equal to the dictionary capacity, and not Count - So I'm returning copies of them taking only 'Count' elements.

1 Like

Is there a newer version of unity that is supporting serializing generic classes now?

Unity 5 maybe (I'm still on Unity 4.6.1f1)?

[edit]
ignore that, you answer my question in there. Just didn't read it all...

This is EXCELLENT. Well done!

I had been using custom classes similar to the first two. Thanks very much!

I haven't read through the class thoroughly yet.

But couldn't a lot of the hash information be generated on deserialize? Implement ISerializationCallbackReceiver and do the leg work there. Reducing the need for all those fields being serialized, thus reducing the serialized size of the dictionary.

The keys and values are still in paired indexes in their respective arrays. So really... the pairing is implied just by that, meaning really only those 2 arrays would have to be serialized. Everything else can be recalculated based on that assumption.

Also, the IEqualityComparer, since it's readonly, becomes useless. It can't be serialized, and can only be set during the constructor. This means it's going to always be null after unity deserializes it. Either it should be removed, or allow it to be changed, so the user can set the comparer at any time (after deserialization).

@lordofduct you're right about IComparer, just something I forgot about. One thing we could do besides making it public is add a serialized string and assign it the assembly qualified name of the user-specified comparer - Then we could write something like:

[SerializeField] string _comparerTypeName;
private IEqualityComparer<TK> _comparer;
private IEqualityComparer<TK> _Comparer
{
       set { _comparer = value; }
       get {
            if (_comparer == null)
            {
                if (string.IsNullOrEmpty(_comparerTypeName))
                    _comparer = IEqualityComparer<TK>.Default;
                else _comparer = Activator.CreateInstance(Type.GetType(_comparerTypeName)) as IEqualityComparer<TK>
            }
            return _comparer;
       }
}

public ctor (IEqualityComparer comparer..)
{
      Initialize .. etc
      if (comparer == null)
      {
           _comparerTypeName = null;
            _Comparer = EqualityComparer<TKey>.Default;
      }
      else
      {
           _comparerTypeName = comparer.GetType().AssemblyQualifiedName;
            _Comparer = comparer;
      }
}

You know something along those lines could work...

About serializing the fields, it would be ideal to serialize only the keys/values yes.But the point was not to use ISerializationCallbackReceiver avoiding the problems mentioned in the links first mentioned in the first page ^
But I will give it a try, one thing though we'd have to be more familiar with the code to see how we'd construct the rest of the data purely from the keys/values, like the _Next array for example.

Yeah, you can do that.

That's actually the basis behind my 'TypeReference' class:
https://github.com/lordofduct/spacepuppy-unity-framework/blob/master/SpacepuppyBase/TypeReference.cs
propertydrawer:
https://github.com/lordofduct/spacepuppy-unity-framework/blob/master/SpacepuppyBaseEditor/Base/Inspectors/TypeReferencePropertyDrawer.cs

As for the links you refer to. What's going on there is a threading issue combined with the serializer. And locks won't work because the unity serializer doesn't attempt locks (a lock doesn't prevent access to an object, it causes another call to lock on the same object to block). The classes in those examples inherit from their respective types (Dictionary, HashSet, etc). These classes are flagged in the .Net/Mono framework as being serializable with the very same SerializableAttribute. This tells unity to go through the fields of that class and check if they need to be serialized. SO, while your callback is going on it might be looking into the fields of the parent class... here's the thing. Unity uses the System.Serializable and the System.NonSerialized attributes, but it ALSO uses its special UnityEngine.SerializeField attribute. So Dictionary and HashSet don't use this attribute... and here's the annoying factor. Unity SAYS that private fields aren't serialized, and that you don't have to tag them with the System.NonSerializedAttribute... but they fail to mention they still touch those fields, for whatever reason. And since .Net/mono has them flagged for .Net/mono style serialization, they're NOT set as System.NonSerialized.

In your class though, those fields you have direct control over. You can flag them as System.NonSerialized, which signals to the unity serializer to not even bother with them at all. So they won't get touched during the callback.

Of course, you may want to continue serializing those fields for speed sake. Recalculating on the callbacks creates that overhead. And since any call to things like 'Instantiate' actually perform a serialize/deserialize, this means any time you clone a GameObject with a script that has this dictionary on it, it takes on that overhead. So it's about balancing the speed vs the memory...

Honestly, I REALLY hate the way that unity implemented there serialization callback. It still relies on fields in the object. It means that you now need a field for the actual value, and a field for the serialized representation of it, that persists through runtime after serialization.

If they just implemented a serialization callback more similar to the .Net ISerializable interface, where they hand you a representation of the serialized data that can have entries read and wrote to, all of this would be avoided.

I have this feeling that just like a lot of their other design choices that they think lay users may find this confusing or hard to use. But as a result we end up with a system that has "magic" behaviour, and memory overhead that the lay user is mostly left ignorant of.

4 Likes

Oh, and I have ran into these same exceptions as well. So in the cases where it can't be avoided, you have to use it and you don't have full control of everything. Here's what I do.

1) create wrappers when possible, I know... this may sound annoying, but do it where it's not a total chaos fest.

2) ALWAYS mark things SerializeField or System.NonSerialized, this is regardless of using the ISerializationCallbackReceiver interface. Just ALWAYS do it, private/public/internal/protected fields alike. Be VERY explicit about your intent, the serializer seems to obey these attributes adequately.

3) Modifying fields flagged as not serialized, you're pretty safe to do whatever with them (as long as they themselves don't access the unity API). But when modifying serialized properties, do them in one fail swoop. Don't enumerate over a field NOT flagged as NonSerialized, use arrays that are just set.

For example this is how I'd serialize a dictionary:

    void ISerializationCallbackReceiver.OnBeforeSerialize()
    {
        _keys = this.Keys.ToArray();
        _values = this.Values.ToArray();
    }

Where _keys and _values are arrays.

The only sucky thing about the arrays is it generates a lot of garbage during serialization calls (at runtime this would be cloning).

@lordofduct Interesting, thanks for the info! I've always wondered why I sometimes see NonSerialized on private fields...

You could Lists instead of arrays ^ for your keys/values. Just clear them in OnBeforeSerialize and iterate over the dictionary manually via GetEnumerator and MoveNext (although you're doing the same thing as foreach, but for some reason foreach generates garbage when used to iter dictionaries, at least on my machine..)

Now how are we to go about reconstructing some of the stuff like _Next, _Buckets and _Version if we don't serialize them?

I'm not sure about _Version, the only usage I found was so that if you ever modify the dictionary while you're enumerating it you'd get exceptions. Maybe we don't even need to reconstruct it and start from 0?

Awesome work!
Thank you very much!

First off, thank you for posting this! I was successful with SerializableDictionary, however I run into issues when I try to use SerializableDictionary when used in conjunction with the SerializableDictionaryDrawer. It seems to be a problem with reflection. Here is the Debug Error Log:

ArgumentException: Object type SerializableDictionary2[System.Int32,System.Single] cannot be converted to target type: SerializableDictionaryIntVector3
Parameter name: val
System.Reflection.MonoField.SetValue (System.Object obj, System.Object val, BindingFlags invokeAttr, System.Reflection.Binder binder, System.Globalization.CultureInfo culture) (at /Users/builduser/buildslave/mono-runtime-and-classlibs/build/mcs/class/corlib/System.Reflection/MonoField.cs:133)
System.Reflection.FieldInfo.SetValue (System.Object obj, System.Object value) (at /Users/builduser/buildslave/mono-runtime-and-classlibs/build/mcs/class/corlib/System.Reflection/FieldInfo.cs:150)
SerializableDictionaryDrawer
2[System.Int32,System.Single].CheckInitialize (UnityEditor.SerializedProperty property, UnityEngine.GUIContent label) (at Assets/Scripts/Editor/SerializableDictionaryDrawer.cs:120)
SerializableDictionaryDrawer`2[System.Int32,System.Single].GetPropertyHeight (UnityEditor.SerializedProperty property, UnityEngine.GUIContent label) (at Assets/Scripts/Editor/SerializableDictionaryDrawer.cs:20)
UnityEditor.PropertyDrawer.GetPropertyHeightSafe (UnityEditor.SerializedProperty property, UnityEngine.GUIContent label) (at C:/buildslave/unity/build/Editor/Mono/ScriptAttributeGUI/PropertyDrawer.cs:37)
UnityEditor.PropertyHandler.GetHeight (UnityEditor.SerializedProperty property, UnityEngine.GUIContent label, Boolean includeChildren) (at C:/buildslave/unity/build/Editor/Mono/ScriptAttributeGUI/PropertyHandler.cs:208)
UnityEditor.EditorGUI.GetPropertyHeightInternal (UnityEditor.SerializedProperty property, UnityEngine.GUIContent label, Boolean includeChildren) (at C:/buildslave/unity/build/Editor/Mono/EditorGUI.cs:4803)
UnityEditor.EditorGUI.GetPropertyHeight (UnityEditor.SerializedProperty property, UnityEngine.GUIContent label, Boolean includeChildren) (at C:/buildslave/unity/build/artifacts/generated/common/editor/EditorGUIBindings.gen.cs:746)
UnityEditor.Editor.GetOptimizedGUIBlockImplementation (Boolean isDirty, Boolean isVisible, UnityEditor.OptimizedGUIBlock& block, System.Single& height) (at C:/buildslave/unity/build/artifacts/generated/common/editor/EditorBindings.gen.cs:204)
UnityEditor.GenericInspector.GetOptimizedGUIBlock (Boolean isDirty, Boolean isVisible, UnityEditor.OptimizedGUIBlock& block, System.Single& height) (at C:/buildslave/unity/build/Editor/Mono/Inspector/GenericInspector.cs:14)
UnityEditor.InspectorWindow.DrawEditor (UnityEditor.Editor editor, Int32 editorIndex, Boolean forceDirty, System.Boolean& showImportedObjectBarNext, UnityEngine.Rect& importedObjectBarRect, Boolean eyeDropperDirty) (at C:/buildslave/unity/build/Editor/Mono/Inspector/InspectorWindow.cs:1124)
UnityEditor.InspectorWindow.DrawEditors (UnityEditor.Editor[ ] editors) (at C:/buildslave/unity/build/Editor/Mono/Inspector/InspectorWindow.cs:969)
UnityEditor.InspectorWindow.OnGUI () (at C:/buildslave/unity/build/Editor/Mono/Inspector/InspectorWindow.cs:350)
System.Reflection.MonoMethod.Invoke (System.Object obj, BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[ ] parameters, System.Globalization.CultureInfo culture) (at /Users/builduser/buildslave/mono-runtime-and-classlibs/build/mcs/class/corlib/System.Reflection/MonoMethod.cs:222)

Any ideas would be welcome! :)

It says right there in the top line of your error.
Object type SerializableDictionary`2[System.Int32,System.Single] cannot be converted to target type: SerializableDictionaryIntVector3

You're trying to cast a Dictionary to a Dictionary, somewhere.

I have a question, in your prime number array, some numbers are missing like 5,7,13,19... is it intentional?
this list is from wikipedia for prime numbers bellow 1000, as you see there are many missing numebrs in your list

2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89, 97, 101, 103, 107, 109, 113, 127, 131, 137, 139, 149, 151, 157, 163, 167, 173, 179, 181, 191, 193, 197, 199, 211, 223, 227, 229, 233, 239, 241, 251, 257, 263, 269, 271, 277, 281, 283, 293, 307, 311, 313, 317, 331, 337, 347, 349, 353, 359, 367, 373, 379, 383, 389, 397, 401, 409, 419, 421, 431, 433, 439, 443, 449, 457, 461, 463, 467, 479, 487, 491, 499, 503, 509, 521, 523, 541, 547, 557, 563, 569, 571, 577, 587, 593, 599, 601, 607, 613, 617, 619, 631, 641, 643, 647, 653, 659, 661, 673, 677, 683, 691, 701, 709, 719, 727, 733, 739, 743, 751, 757, 761, 769, 773, 787, 797, 809, 811, 821, 823, 827, 829, 839, 853, 857, 859, 863, 877, 881, 883, 887, 907, 911, 919, 929, 937, 941, 947, 953, 967, 971, 977, 983, 991, 997

also this may help?

https://www.mathsisfun.com/numbers/prime-number-lists.html

[quote=“Jiraiyah”, post:14, topic: 586385]
I have a question, in your prime number array, some numbers are missing like 5,7,13,19… is it intentional?
this list is from wikipedia for prime numbers bellow 1000, as you see there are many missing numebrs in your list

2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89, 97, 101, 103, 107, 109, 113, 127, 131, 137, 139, 149, 151, 157, 163, 167, 173, 179, 181, 191, 193, 197, 199, 211, 223, 227, 229, 233, 239, 241, 251, 257, 263, 269, 271, 277, 281, 283, 293, 307, 311, 313, 317, 331, 337, 347, 349, 353, 359, 367, 373, 379, 383, 389, 397, 401, 409, 419, 421, 431, 433, 439, 443, 449, 457, 461, 463, 467, 479, 487, 491, 499, 503, 509, 521, 523, 541, 547, 557, 563, 569, 571, 577, 587, 593, 599, 601, 607, 613, 617, 619, 631, 641, 643, 647, 653, 659, 661, 673, 677, 683, 691, 701, 709, 719, 727, 733, 739, 743, 751, 757, 761, 769, 773, 787, 797, 809, 811, 821, 823, 827, 829, 839, 853, 857, 859, 863, 877, 881, 883, 887, 907, 911, 919, 929, 937, 941, 947, 953, 967, 971, 977, 983, 991, 997

also this may help?

https://www.mathsisfun.com/numbers/prime-number-lists.html
[/quote]

So the use of primes is only to create an array that is a length that when modulo’d with a hash you get an index with low probability of collision.

This use case only requires that the length is prime… it doesn’t necessarily have to be all primes.

As your collection grows, the odds increase of course. If you have a collection of capacity 11 and 2 elements, very low odds of collision. A collection of capacity 11 and 9 elements, there are high odds of collision. Over 11 and you have guaranteed collision.

So when you get close to the size of the capacity, the array inside your collection, you resize to a length of a larger prime number. Doesn’t have to be the next, just larger.

Because, if that next prime number is only 2 away… like going from 11 to 13. And you were already really close to full, like earlier when I described 9 elements in a collection sized at 11. You haven’t really reduced your risk of collision at all!

Within just a couple adds, you’ll be back to having to resize your array yet again.

Thing is… array resizing is NOT cheap. You want to reduce the frequency of that.

So, you spread out the primes. Creating larger and larger gaps as your collection grows. But too large so you don’t consume unnecessary chunks of memory.

The spacing vexe is using appears to be a fairly standard spacing for hash tables. I’m betting vexe didn’t sit down and calculate those primes themselves, and probably pulled a list from somewhere. And rightfully so… who wants to sit there and calculate a well distributed set of primes every time they go and write something like this… might as reuse a pre-compiled list that has shown to be well distributed.

4 Likes

Oh, and I should point out, looking back on OP code.

I want to repeat that you really shouldn't be serializing all that information, and rather recalculate all of it on deserialize, aside from the key/value arrays.

One reason is that 'hashcodes' aren't supposed to be considered persistent across systems. From MSDN documentation for 'GetHashCode':
[quote]
A hash code is intended for efficient insertion and lookup in collections that are based on a hash table. A hash code is not a permanent value. For this reason:

  • Do not serialize hash code values or store them in databases.

  • Do not use the hash code as the key to retrieve an object from a keyed collection.

  • Do not send hash codes across application domains or processes. In some cases, hash codes may be computed on a per-process or per-application domain basis.

  • Do not use the hash code instead of a value returned by a cryptographic hashing function if you need a cryptographically strong hash. For cryptographic hashes, use a class derived from the System.Security.Cryptography.HashAlgorithm or System.Security.Cryptography.KeyedHashAlgorithm class.

  • Do not test for equality of hash codes to determine whether two objects are equal. (Unequal objects can have identical hash codes.) To test for equality, call the ReferenceEquals or Equals method.
    [/quote]

https://msdn.microsoft.com/en-us/library/system.object.gethashcode(v=vs.110).aspx

1 Like

I got it, thanks for info,
But lordofduct, can you implement your own version without serializing all those fields? I am really new to serialization but in serious need of a serializable dictionary, maybe the OP won't do it any more, so, is it possible that you put a little time and do it? I would appreciate it a lot

Sure...

But I will point out that I would NOT use the technique above. A hashtable would not store compactly at all, and wouldn't travel between app domains well. To get everything, you'd end up storing more data then if you were to just serialize up the key/value pairs.

Actually, even Microsoft agrees, because if you look at their implementation. They only store the key/value pairs and rebuild the table on deserialize.

This is the reason I don't mind slapping this together... I technically have been meaning to do it for quite some time, and also it's not that much work.

There comes another issue though... the inspector for this would be UGLY. So you'll want an inspector that better describes the situation.

The result:

SerializableDictionaryBase:
https://github.com/lordofduct/spacepuppy-unity-framework/blob/master/SpacepuppyBase/Collections/SerializableDictionaryBase.cs
Note that it inherits from 'DrawableDictionary', this is so that we can define a PropertyDrawer that always applies. You can't create a PropertyDrawer that targets the generics version of the class, it only targets concrete classes. So instead we have to do this.

This is also why I have to implement IDictionary, instead of just directly inherit from Dictionary. All the methods in there for the implementation really just forward to the actual dict.

The serialization just turns the dict into arrays.

I would have rather inherited directly from Dictionary... but alas, the limitations with unity serialization and editor extension.

The PropertyDrawer:
https://github.com/lordofduct/spacepuppy-unity-framework/blob/master/SpacepuppyBaseEditor/Collections/DictionaryPropertyDrawer.cs
Honestly, the PropertyDrawer is the more complicated part.

Because it's generic, you have to create concrete classes in your scripts:

using UnityEngine;
using System.Collections;

using com.spacepuppy;
using com.spacepuppy.Collections;

public class TestScript05 : MonoBehaviour {


  [SerializeField()]
  private LocalDictionary _dict;

  public LocalDictionary Dictionary
  {
  get { return _dict; }
  }

  [System.Serializable()]
  public class LocalDictionary : SerializableDictionaryBase<string, int>
  {

  }

}

And it looks like:
2282920--153282--SerializedDict.png

WARNING - I have not done tons of testing. But I can tell you that dictionary's do NOT guarantee order. This means that the order you define the entries in the inspector isn't guaranteed to be the same the next time. Just that the same entries exist.

Because inspectors may serialize/deserialize during drawing, you could actually end up with the order changing on you when you click the add/remove buttons in my inspector.

This is actually why I went with this add/remove technique. The resizable array comes with huge issues... especially when resizing the 'keys' array. Because the keys have to be unique, and resizing arrays in the inspector causes dupes, you can't resize.

I would have made it prettier, the inspector that is, but I didn't want to create a lot of dependencies to the rest of my framework. There still one dependency, and that's to 'EditorHelper.SetPropertyValue', which you can find here:
https://github.com/lordofduct/spacepuppy-unity-framework/blob/master/SpacepuppyBaseEditor/EditorHelper.cs

To anyone in the future, this thing will probably be getting expanded further when I have the time... so there will probably be more dependencies added.

4 Likes

well i should thank you to no end,
1- the order is really not important, normally when someone goes for dictionaries, he is thinking about a collection without specific orders, so that should be ok
2- for the property drawer, did you considered using the reorderable inspector stuff? that is the nicest thing for lists i saw, although it's internal for unity but available for us to use ;)

It is, and I want to, but like I said I wanted to avoid integrating it too tight with my framework. And the way I use ReorderableList in PropertyDrawer would have done so.

It also takes a little bit more work then the simple slapdash one I did here.

I'll be updating it in the future when I have time.