Use the JSON serilization for ScriptableObject with asset fields in Editor and Build

Unity Engine
,
1

I tested the possibility to use the ScriptableObject to Store Game settings.
The default values Stored in asset.

[CreateAssetMenu(fileName ="GameSettingsSO",menuName = "SoundAndEffects/GameSettingsSO")]
public class GameSettingsSO : ScriptableObject
{
    [Header("Game Options")]
    [SerializeField] private ComplexitySO _complexityGame;
    [SerializeField] private PlayMode _usedPlayMode;
    [Header("Audio Options")]
    [SerializeField] private float _masterVolume;
    [SerializeField] private float _musicVolume;
    [SerializeField] private float _effectVolume;
    [SerializeField] private SequenceType _musicSequenceType;

    public override string ToString() => JsonUtility.ToJson(this);

    public PlayMode UsedPlayMode => _usedPlayMode;
    public ComplexitySO ComplexityGame => _complexityGame;
}

The Changed Settings saved to disk through JSON serialization as string
JsonUtility.ToJson(storeScriptableObjectData);
Was detected strange behavior:
EDIT see note at next post

When I store in Play mode in Editor the field “_complexityGame” serialized like as :{“instanceID”:18942}

{"_complexityGame":{"instanceID":18942},"_usedPlayMode":2,"_notCopyToGlobal":true,"_globalDefaultTopList":false,"_masterVolume":1.0,"_musicVolume":2.0,"_effectVolume":3.0,"_musicSequenceType":1}

When I store in Play mode in Build the field “_complexityGame” serialized like as :{“m_FileID”:2154,“m_PathID”:0}

{"_complexityGame":{"m_FileID":2154,"m_PathID":0},"_usedPlayMode":2,"_notCopyToGlobal":true,"_globalDefaultTopList":false,"_masterVolume":1.0,"_musicVolume":2.0,"_effectVolume":3.0,"_musicSequenceType":1}

Every time it was contained the same value

I can understand the reason of that, and for save setting it’s not a big problem (you may not transfer this to build).
But if you try to save something in processing of developing (as JSON) and how it can restore after in build, in case if it will not primitive values?

2

The short answer is you can’t serialise out reference to other UnityEngine.Object assets, even using Unity’s JSON Utility and restore them later.

3

I understand it now, but it was a little unexpected :).
I plan to test the Odin JSON Serialization, or another variants, but they all based on fact that you write - exist the limitation of standard JSON Unity Serialization, and use the custom JSON Serialization ( ISerializationCallbackReceiver ).

But the main problem of this issue with serialization (different algorithms of serialization links to assets in Editor and in Build) you detects it only if will use it. I don’t remember that it was described.

And that more interesting why Unity doesn’t use the same algorithms of serialization of links to assets in Editor and in Build) ({"instanceID":18942} vs {"m_FileID":2154,"m_PathID":0})??

4

Well, Unity does not really have a proper representation of the AssetDatabase at runtime. It’s essentially an editor feature for the authoring of your game. In the editor you only have asset references. So instances that are stored in the asset database. A single serialization unit may also use local FileIDs to refer to other instances within the same serialization unit. Unity uses YAML or it’s custom binary asset format to store this information. The json serialization does not include FileIDs since json itself does not have a concept for that. Some json serializers do implement meta tags to store such information and technically that’s of course possible. Though Unity’s serialization system has grown quite “organically” over many years and it gets progressively harder to make radical changes.

At runtime we only have instance IDs. Instance IDs are not globally unique and are usually restricted to a single serialization unit (scene, perfab, asset). I guess it’s quite difficult to change the current system to be actually useful for runtime serialization.

Don’t get me wrong, I would also love to see a better assetdatabase access at runtime. Of course only read-only access. Though simply getting a GUID of an authored asset at runtime and looking up an asset based on that GUID would make it extremely easy to roll your own serialization. We now have the addressables system which is kind of a hack on top of the existing system. It still does not fill all the gaps. In order to be able to create a proper save system we simply need a way to identify assets stored in the asset database at runtime and we need a way to address those later. Likewise the SerializeReference attribute (which gives us a lot more flexibility) is only restricted to the serialization unit and can’t be used for cross references. We have many tiny tools which all improve what we can do, but the above mentioned basic features are still missing in Unity.

While I never really wanted a ready-to-use save and load solution from Unity like most other major engines may have, I prefer some simple basic features where we can build on top our own solutions. Most ready-to-use solutions have limitations or specific game types in mind (games like the old halflife actually store the whole bsp map with all dynamic changes in the save file. So loading a savegame does only partly load the original map). Unity is supposed to be the “allround” solution for games. It should not be stuck at a certain game type. Different games have vastly different save and load needs. So having basic tools to interact with the engine and its assets would be the best solution. We don’t need a complicated and convoluted solution that tries to satisfy all needs and usually fail reaching that goal. We just need the basic tools. The individual solutions will be developed by the community / developers.

That was an attempt to defend Unity and at the same time a rant why we still don’t have the tools we need :slight_smile:

5

You cannot, I repeat CANNOT serialise out Unity objects or references to them nor can you restore those. DO NOT waste your time trying to do so.

Trust me I bought Odin with the intention of serialising out scriptable objects to learn that it couldn’t. Upside is that I ended up with an awesome addon that does do a million other things.

Your only choice is to devise a system to represent references to unity objects and the objects themselves with simple, serialisable forms. Basically with plain classes and structs, often ones that are ‘serialisable surrogates’ of your Unity centric types. There is no getting around this.

6

Right, it’s sad that we have to do this, but currently it’s the only real solution. Depending on the needs you essentially have to roll your own “asset database” of objects you want / need to reference and give them your own ID which you can look up. either simply an index into an array of objects or some other solution. Depending on the game you have to decide which granularity you need.

7

Thanks you @spiney199 and @Bunny83 for explanation the existent limitations and some issues with Unity Serialization.

Note
Also I must noted :slight_smile: that I was initially wrong, because when I tested the serialize the objected with a asset reference in Editor :slight_smile: I forgot to reloading the Editor self between Save and Restore (I was reloading Scene only).
In this the case (the “instanceID” not changing :slight_smile: as described it in Unity documentaion) and the Store& Restore of reference to asset was restoring “correctly”. Now i see that is not possible without use a some workarounds and explanations from @spiney199 and @Bunny83

8

We trivially work around this as follows:

  • we recognize that ScriptableObjects cannot be serialized.

  • we put ALL ScriptableObjects of a given type in a single directory to enforce name uniqueness

  • we make a Repository object to retrieve these ScriptableObjects by name

  • we modify our serializable structures so they:
    ----> don’t serialize the SO itself
    ----> they serialize the name of the SO
    ----> they load the SO as a reference and look it up

You cannot use Unity’s “lite” JSON implementation to do this.

The following example uses Newtonsoft JSON, available via the Unity Package Mangler:

    [System.Serializable]
    public class MySerializableThingamajig
    {
        // this is what we drag into the inspector
        [Newtonsoft.Json.JsonIgnore]
        public VisualizationSO Visualization;

        // being a property, this doesn't appear in the inspector
        // being a proper JSON mechanism, NewtonSoft handles it correctly
        public string VizualizationName
        {
            get
            {
                return Visualization.name;
            }
            set
            {
                Visualization = VisualizationSORepository.Load(value);
            }
        }
    }

This also has the side benefit of using an easy-to-reason-about name in the save data, something you can look up directly in the project rather than some GUID or instance ID or other ambiguous number.

The big obvious hole here is if that name is serialized, the underlying ScriptableObject that the name references can:

  • change
  • become deleted
  • become invalid

But since we control the repository of these objects, we can:

  • simply elect to invalidate old names, or
  • provide lookup service to legacy names, or
  • otherwise have appropriate graceful behavior take place when stale data is deserialized.

A serialization data format specifier can be useful here, such as a version number.

9

This is why I impose a unique identifier inside of SO’s that I need to save out. Easy enough to make a small Utility class that generates a Guid when the SO is made, and use it to both save out and look up from a dictionary.

I would post my example but you’d probably murder me for how over engineered it is. ;p

10

and you call it by VisualizationSORepository.Load(value);

and I didn’t dive to load an assets in runtime (Asset Bundles/Addressable and so on) … did understand right - you propose to base the VisualizationSORepository.Load on methods Resources.Load

11

I will not kill you :slight_smile: exactly, because for me interesting your concept also, because when i thinking about surrogate for SO serialization (i thought about the same).
OFFTOP
But that will do the Wizard of 27K Level i don’t know:eyes: … but i think he love the animals :slight_smile: and may be peoples also:) (therefore you have a double chance :))

12

Well I was replying to Kurt there, but the code also uses a bit of Odin Inspector. Nonetheless, here’s the utility class I use:

namespace LizardBrainGames
{
    using System;
    using UnityEngine;
    using Sirenix.OdinInspector;

    /// <summary>
    /// Serializable class that handles the storing and generation of a GUID string.
    /// Supports implicit conversion to a string.
    /// </summary>
    [Serializable, HideLabel]
    public sealed class AssetGUID : IEquatable<AssetGUID>
    {
        #region Constructors

        /// <summary>
        /// Creates a new AssetGUID instance. If generateGUID is true, a new GUID will be generated immediately.
        /// </summary>
        public AssetGUID(bool generateGIUD = false)
        {
            if (generateGIUD == true) GenerateGUID();
        }

        #endregion

        #region Inspector Fields

        [HorizontalGroup("GUID", Width = 0.5f, MaxWidth = 1)]
        [SerializeField, HideLabel, ReadOnly]
        private string assetGUID;

        #endregion

        #region Properties

        /// <summary>
        /// Returns the asset GUID as a string.
        /// </summary>
        public string AssetGUIDString { get { return assetGUID; } }

        /// <summary>
        /// Does this AssetGUID already have a generated GUID?
        /// </summary>
        public bool HasGeneratedGUID { get { return !string.IsNullOrEmpty(assetGUID); } }

        #endregion

        #region GUID Generation Methods

        /// <summary>
        /// Call this to generate a new GUID. Will only generate a new GUID if none is already generated.
        /// </summary>
        [HorizontalGroup("GUID")]
        [Button("Generate GUID"), ShowIf("@!HasGeneratedGUID")]
        public void GenerateGUID()
        {
            if (!HasGeneratedGUID) assetGUID = Guid.NewGuid().ToString();
        }

        /// <summary>
        /// Generates a new GUID even if one already exists.
        /// </summary>
        public void ForceGenerateGUID()
        {
            assetGUID = Guid.NewGuid().ToString();
        }

        #endregion

        #region Interface Methods & Object Overloads

        /// <summary>
        /// Compares if two AssetGUID classes contain the same GUID string.
        /// </summary>
        public bool Equals(AssetGUID other)
        {
            if (other is null) return false;

            if (ReferenceEquals(this, other)) return true;

            return assetGUID == other.assetGUID;
        }

        public override bool Equals(object obj)
        {
            return obj is AssetGUID assetGUID && this.Equals(assetGUID);
        }

        public override int GetHashCode()
        {
            return assetGUID.GetHashCode();
        }

        public override string ToString()
        {
            return assetGUID;
        }

        #endregion

        #region Operator Overloads

        /// <summary>
        /// Returns true if two Asset GUID classes contain the same GUID string.
        /// </summary>
        public static bool operator ==(AssetGUID left, AssetGUID right)
        {
            return left is null ? right is null : left.Equals(right);
        }

        /// <summary>
        /// Returns true if two asset GUID classes do not contain the same GUID string.
        /// </summary>
        public static bool operator !=(AssetGUID left, AssetGUID right)
        {
            return !(left == right);
        }

        /// <summary>
        /// Returns true if the string equals the AssetGUID's GUID string.
        /// </summary>
        public static bool operator ==(string text, AssetGUID assetGUID)
        {
            return assetGUID.assetGUID == text;
        }

        /// <summary>
        /// Returns true if the string does not equal the AssetGUID's GUID string.
        /// </summary>
        public static bool operator !=(string text, AssetGUID assetGUID)
        {
            return !(text == assetGUID);
        }

        /// <summary>
        /// Returns true if the string equals the AssetGUID's GUID string.
        /// </summary>
        public static bool operator ==(AssetGUID assetGUID, string text)
        {
            return text == assetGUID;
        }

        /// <summary>
        /// Returns true if the string does not equal the AssetGUID's GUID string.
        /// </summary>
        public static bool operator !=(AssetGUID assetGUID, string text)
        {
            return !(text == assetGUID);
        }

        /// <summary>
        /// Returns the Asset GUID string with an implicit conversion.
        /// </summary>
        public static implicit operator string(AssetGUID assetGUID)
        {
            return assetGUID.assetGUID;
        }

        #endregion
    }
}

And I use an interface to make it more of a contract:

namespace LizardBrainGames
{
    /// <summary>
    /// Interface for assets to implement that want to express a unique identifier.
    /// </summary>
    public interface IAssetGUID
    {
        /// <summary>
        /// Accessor to the asset's AssetGUID data.
        /// </summary>
        public AssetGUID AssetGUID { get; }
    }
}

Most likely they just have simple databases of their own. I just use, you guessed it, more scriptable objects as simple databases.

13

Spiney! I would never murder you! But I will say that since the filesystem imposes unique names, I’m happy to lean on that technology. Your mileage may vary. I just love code that doesn’t exist because then it cannot have bugs. :slight_smile:

14

I did indeed base it upon that, but it could be based on any resource locator pattern you want: Resources.Load(), Addressables, your own flavor, Spiney’s GUID-based type, etc.

15

Summary:
The current Unity architecture doesn’t give a possibility to serialize any objects (instance of class) (e.g. ScriptableObject) with Type fields with reference to any Unity assets (include other ScriptableObject), especial in build of Game.
Because the current main API to work with assets in runtime is the methods of class Resources (UnityEngine namespace) which give only possibility to load asset based on known path to the asset. And doesn’t give a possibility to detect the path of the loaded asset.
In this case to store this Type objects (in Runtime especial) must be used any workarounds described in this thread by @Kurt-Dekker and @spiney199 .
But to be completely honest, if you want store this Type objects in Editor you can use the Odin Serializer (free part software of Odin Inspector), which use the possibilities the methods of class AssetDatabase (UnityEditor namespace !!!).
The TeamSirenix made the good products, but not very clear documentations (the procedure partially described on they GitHub TeamSirenix and partially on they tutorials External Reference Resolver, below I put the compilation from these sources, because a part of the original code contains many small weird errors):

  • The object to store is GameSettingsSO (described in my first post here ).

  • The procedure of Store contains 2 steps (the Load will use the antiwise sequence of steps)):

  • Call Save and receive the file with references to assets and List unityReferences which contains list of these assets.

  • Call one method (here the SaveGuiDRef) from “External Reference Resolver” (the methods which realize the IExternalStringReferenceResolver or IExternalGuidReferenceResolver interface) which for every element from the list get the GuiD of asset and store it in the separate file.

using System.Collections.Generic;
using UnityEngine;
using OdinSerializer;
using System.IO;

public static class Example
{
    private const DataFormat FormatData = DataFormat.JSON;

    public static void Save(GameSettingsSO data, string filePath, ref List<Object> unityReferences)
    {
        byte[] bytes = SerializationUtility.SerializeValue(data, FormatData, out unityReferences);
        File.WriteAllBytes(filePath, bytes);
    }

    public static GameSettingsSO Load(string filePath, List<Object> unityReferences)
    {
        byte[] bytes = File.ReadAllBytes(filePath);
        return SerializationUtility.DeserializeValue<GameSettingsSO>(bytes, FormatData, unityReferences);
    }

    static byte[] Serialize(object obj, SerializationContext context)
    {
        return SerializationUtility.SerializeValue(obj, FormatData, context);
    }

    static object Deserialize(byte[] bytes, DeserializationContext context)
    {
        return SerializationUtility.DeserializeValue<object>(bytes, FormatData, context);
    }

   public static void SaveGuiDRef(string filePath, UnityEngine.Object unityReferences)
    {
        var context = new SerializationContext()
        {
            GuidReferenceResolver = new ScriptableObjectGuidReferenceResolver(),
        };
        byte[] bytes = Serialize(unityReferences, context);
        File.WriteAllBytes(filePath, bytes);
    }

    public static Object LoadGuiDRef(string filePath)
    {
        byte[] bytes = File.ReadAllBytes(filePath);
        var context = new DeserializationContext()
        {
            GuidReferenceResolver = new ScriptableObjectGuidReferenceResolver(),
        };
        return (Object)Deserialize(bytes, context);
    }
}

I removed little errors and refactored the ScriptableObjectGuidReferenceResolver w/o the “ToString(“N”)” it is not work initially.
Didn’t test the ScriptableObjectStringReferenceResolver, but with the same refactoring it will also works

public class ScriptableObjectGuidReferenceResolver : IExternalGuidReferenceResolver
{
    public IExternalGuidReferenceResolver NextResolver { get; set; }

    public bool CanReference(object value, out Guid id)
    {
        if (value is ScriptableObject)
        {
            id = new Guid(AssetDatabase.AssetPathToGUID(AssetDatabase.GetAssetPath(value as ScriptableObject)));
            return true;
        }

        id = default(Guid);
        return false;
    }

    public bool TryResolveReference(Guid id, out object value)
    {
        value = AssetDatabase.LoadAssetAtPath<ScriptableObject>(AssetDatabase.GUIDToAssetPath(id.ToString("N")));
        return value != null;
    }
}

As you can see, it uses methods from the AssetDatabase class, so it won’t work in the Build.

16

To make the output of these scripts more visual, I replaced the GameSettingSO on this class (It’s a plain c# class with one field with ref to asset - ComplexitySO, don’t forget the output Odin JSON little differ from std Unity JSON):

public class GameSettings
{
    [SerializeField] private ComplexitySO _complexityGame;
    [SerializeField] private PlayMode _usedPlayMode;
    [SerializeField] private bool _notCopyToGlobal;
    [SerializeField] private bool _globalDefaultTopList;
    [SerializeField] private float _masterVolume;
    [SerializeField] private float _musicVolume;
    [SerializeField] private float _effectVolume;
    [SerializeField] private SequenceType _musicSequenceType;
}

and put the result of output after first step:

{
    "$id": 0,
    "$type": "0|GameSettings, Assembly-CSharp",
    "_complexityGame": $eref:0,
    "_usedPlayMode": 2,
    "_notCopyToGlobal": true,
    "_globalDefaultTopList": true,
    "_masterVolume": 2,
    "_musicVolume": 3,
    "_effectVolume": 4,
    "_musicSequenceType": 1
}

and after Second:

$guidref:a385e0b8-1915-59e4-5bd9-a781a116b3ba
17

Still sounds like you’re still at square one, honestly. As Bunny said, you still have to produce your own lookup system to convert these GUID’s back to asset references.

I’ve been working on just that myself, but in a manner which should hopefully be project agnostic. It’ll also use the Odin Serialiser’s external reference resolvers, but swap out assets with various types of substitutes and restore them from a lookup table system.

18

:slight_smile: I simple love make some summary, which i can think may be useful for others (and I two days made a rest - was helped to Agamemnon to defeat Troy (ToW))
… And I know why on forum used the the Surrogate term to discuss the solutions for these serialization issues.

19

I guess this was just a quickly written example, because this would not be valid json and could not be read by any json parser :slight_smile: You probably wanted to quote that value $eref:0, right?

Though as Spiney has reiterated, Unity doesn’t really provide any kind of id that can be used for read / write operations. So you have to come up with your own system for giving objects ids and a way to search for them / look them up.

20

No, It’s a real output from code (copy/past) which I received when I tested the Save/Load the Object with ref to asset in Editor (and show in this post)… and it work correctly (it could be Odin’s internal JSON format).
Because the every value placed between <:> <,>, it’s not created problem for parsing.
And i think if it will be demands exist the option which can change the format JSON to standard, but in this format it more comfortable to read by human (in the standard so many quotes)