I just came up with an improvement idea for Unity Properties, not sure if it’s feasible.
Anyway, I checked the generated code by Unity Properties, for non-public fields, it uses Reflection and Emit to implement GetValue and SetValue. I was thinking, perhaps we can place the generated property class inside the container type, like this:
// User Type
[GeneratePropertyBag]
partial class Container
{
[CreateProperty(insidePartial: true)]
int _field;
}
// Generated
partial class Container
{
public class _field_XXXX_Property : Property<Container, int>
{
public override string Name => "_field";
public override bool IsReadOnly => false;
public override int GetValue(ref Container container)
=> container._field;
public override void SetValue(ref Container container, int value)
=> container._field = value;
}
}
Firstly, CreateProperty could allow receiving a parameter insidePartial. If set to true, the generated property class would be located inside the container, enabling direct access to non-public members of the container without the need for Reflection and Emit.
The only issue is that the container may not have the partial modifier. In such cases, if insidePartial is true, CreateProperty would check whether the container type, and the types containing it all have the partial modifier. If not, it would set insidePartial to false and output a warning, to reminding the user to consider adding the partial modifier.
Finally, regardless of whether this idea is feasible, fields with the internal modifier should be accessible directly, similar to public fields, without the need for Reflection and Emit.
Hi @yu_yang , if you add the partial keyword to the type tagged with [GeneratePropertyBag], the property bag will be generated as a nested type. For example, this:
[assembly:Unity.Properties.GeneratePropertyBagsForAssembly]
[Unity.Properties.GeneratePropertyBag]
public partial class MyClass
{
public float value;
}
Should output something like:
partial class MyClass
{
internal static void RegisterMyClass_7f11a2605bf9464c8f93f66930bb242a_PropertyBag()
{
global::Unity.Properties.PropertyBag.Register(new global::MyClass.MyClass_7f11a2605bf9464c8f93f66930bb242a_PropertyBag());
}
[global::System.Runtime.CompilerServices.CompilerGenerated]
sealed class MyClass_7f11a2605bf9464c8f93f66930bb242a_PropertyBag : global::Unity.Properties.ContainerPropertyBag<global::MyClass>
{
public MyClass_7f11a2605bf9464c8f93f66930bb242a_PropertyBag()
{
AddProperty(new value_Property());
}
[global::System.Runtime.CompilerServices.CompilerGenerated]
class value_Property : global::Unity.Properties.Property<global::MyClass, float>
{
public override string Name => "value";
public override bool IsReadOnly => false;
public override float GetValue(ref global::MyClass container) => container.value;
public override void SetValue(ref global::MyClass container, float value) => container.value = value;
}
}
}
It doesn’t go into great details about it, but this page indicates the making type partial will allow the property bag to access internal and private fields and properties.
Wait a moment, the situation seems a bit off. My test code is like this:
[GeneratePropertyBag]
partial class Container
{
[CreateProperty]
int _field;
[GeneratePropertyBag]
partial class Container2
{
[CreateProperty]
int _field;
}
}
But the generated code is like this (Please note that the property type still inherits from ReflectedMemberProperty):
[GeneratePropertyBag]
internal class Container
{
[CreateProperty]
private int _field;
internal static void RegisterContainer_f34c865745ed49bb85d97611bb05bfac_PropertyBag()
{
PropertyBag.Register<Container>((PropertyBag<Container>) new Container.Container_f34c865745ed49bb85d97611bb05bfac_PropertyBag());
}
internal static void RegisterContainer2_da2e8613d194467f998744afe5a074b1_PropertyBag()
{
Container.Container2.RegisterContainer2_da2e8613d194467f998744afe5a074b1_PropertyBag();
}
public Container()
{
base.\u002Ector();
}
[GeneratePropertyBag]
private class Container2
{
[CreateProperty]
private int _field;
internal static void RegisterContainer2_da2e8613d194467f998744afe5a074b1_PropertyBag()
{
PropertyBag.Register<Container.Container2>((PropertyBag<Container.Container2>) new Container.Container2.Container2_da2e8613d194467f998744afe5a074b1_PropertyBag());
}
public Container2()
{
base.\u002Ector();
}
[CompilerGenerated]
private sealed class Container2_da2e8613d194467f998744afe5a074b1_PropertyBag :
ContainerPropertyBag<Container.Container2>
{
public Container2_da2e8613d194467f998744afe5a074b1_PropertyBag()
{
base.\u002Ector();
this.AddProperty<int>((Property<Container.Container2, int>) new Container.Container2.Container2_da2e8613d194467f998744afe5a074b1_PropertyBag._field_Property());
}
[CompilerGenerated]
private class _field_Property : ReflectedMemberProperty<Container.Container2, int>
{
public _field_Property()
{
base.\u002Ector(typeof (Container.Container2).GetField("_field", BindingFlags.Instance | BindingFlags.NonPublic), "_field");
}
}
}
}
[CompilerGenerated]
private sealed class Container_f34c865745ed49bb85d97611bb05bfac_PropertyBag :
ContainerPropertyBag<Container>
{
public Container_f34c865745ed49bb85d97611bb05bfac_PropertyBag()
{
base.\u002Ector();
this.AddProperty<int>((Property<Container, int>) new Container.Container_f34c865745ed49bb85d97611bb05bfac_PropertyBag._field_Property());
}
[CompilerGenerated]
private class _field_Property : ReflectedMemberProperty<Container, int>
{
public _field_Property()
{
base.\u002Ector(typeof (Container).GetField("_field", BindingFlags.Instance | BindingFlags.NonPublic), "_field");
}
}
}
}
I’m not certain if this is a version issue. My Unity version is 2022.3.14f1.
Not a version thing, this is a bug in the source generator. It should be able to generate code that does not use reflection when it’s generated as a nested type.
I’ll get that fixed once I’m back from the holidays!
After testing, it was found that after calling ToBinary, modifying the order of fields in the code, and then calling FromBinary, the object cannot be correctly restored. Is it because there are no names for serialized properties? If so, it would be best to provide an option to serialize property names, with the default being true, to ensure that data can be correctly restored after version changes.
From what i’ve seen, currently only the Json adapter has the default Unity behaviour, where “renamed” properties marked with [FormerName] will get picked up. Also, anything more complex, still on Json, you can implement IJsonMigration which is great.
I dont think they will implement what you mean for Binary because the non-versioned behaviour is actually quite logical.
Json is a structured format, any serialized data makes sense on its own. Its a collection of key-value properties following javascript’s object notation (JavaScript Object Notation = json). So it doesnt matter where you read it, it will always make sense.
Binary on the other side is not a format, just a way of refering to the output as being bytes. You could write a BinaryJson wrapper, that emits json-formatted data in binary though.
Part of the beauty of the BinarySerialization in this package is the crazy speed that comes by not needing to “format” anything and just read and write class definitions directly thanks to Unity.Properties.
Also, @CodeSmile showed aboved a custom implementation for BinaryAdapters that takes into account versioning, which might be helpful!
Version changes complicate binary serialization. Due to the likelihood of version changes, even if they don’t ultimately occur, it’s necessary to anticipate this possibility. Therefore, binary serialization should provide an appropriate way to handle version changes effectively. Note that it should be simple enough, preferably fully automated (similar to Json serialization), and at least simpler than manually writing a BinaryReader; otherwise, it loses the purpose of using Unity Properties.
Keep in mind that the primary goal of using Unity Properties should be automation, with performance as a secondary consideration. If automation isn’t achievable, custom approaches can be used to trade for performance. Automation, in this context, means being able to work out of the box in most scenarios and consistently in a robust way, which is crucial for real projects.
I dont agree with the “should”, i dont think that is explicitly stated by Unity. And from my POV, Unity.Properties is just an API for object graph visitation, leveraging SourceGenerators to make it extremely performant. The problem you are describing applies only to the the BinaryAdapter implementation of Unity.Serialization.
The package provides 2 ways to serialize any data. The first outputs in a structured universally known format, and the other outputs a extremely compacted binary stream.
Implementing the BinarySerialization with a format that is resilient to class definition changes, means a complete overhaul of the system and a huge increase of output size, which in turn can also lose its purpose for many high.performance use cases.
Consider a simple class with two just Int32 fields. The output, if the target class is known at deserialization time (DisableRootAdapters = true), its just 4bytes per field. Total 8 bytes.
Now consider you use a strong-binary format that is resilitent to changes. For it to be resilient, you need to serialize property names, but then you also need either the property type or the size of the written data, so you know when a property ends and the next starts. Writing the data size seems the simple approach. You are now serializing at least 1 byte for each character in the property name (i dont know the specifics of char as bytes, but i think they can also be of size 2byte per char), plus the string’s total size as another 4bytes (or 2bytes using ushort, or 1 byte if you cap it to 255 max chars, which is very reasonable) , and also 4bytes for the Int32 representing the total written size of the property value, plus the value itself. You are easily doubling, probably way more, the output size. Plus the overhead of preparing a SerializedValueView (like the Json counterpart) for the user to consume.
At least myself, I highly prefer the current binary approach, and If somehow you are constrained to a binary output, and you want a version-resilient format, you could actually just use the “JsonSerialization.ToJson” overload that takes in the JsonWriter. Then, instead of converting the JsonWriter to a string, just use the GetUnsafeReadOnlyPtr() method + Length to write into a binary stream. Then for deserialization, you create a SerializedObjectReader from that binary stream, and use that into the JsonSerialization.FromJson(), and voilá. Also, as your output is binary and you wont care about it being human-readable, you could disable StringEscapeHandling, and enable Simplified and Minified in the JsonSerializationParameters, which will be way faster than traditional json)
It seems there is no built-in Clone method? Since the properties have already been collected in advance, I think an efficient cloning method can be implemented without resorting to serialization/deserialization.
BTW, In non-main threads, is it safe to use FromJson/FromBinary?
Here is a full example for the player class I used for the serialization example:
using System;
using Unity.Serialization.Json;
using UnityEditor;
using UnityEngine;
public struct int2
{
public float x;
public float y;
public int2(float x, float y)
{
this.x = x;
this.y = y;
}
}
public enum ItemType
{
Weapon,
Armor,
Consumable
}
public class Item
{
public string Name;
public ItemType Type;
}
public class Player
{
public string Name;
public int Health;
public int2 Position;
public Item[] Inventory;
}
public static class Test
{
[InitializeOnLoadMethod]
public static void RunTest()
{
JsonSerialization.AddGlobalAdapter(new Adapter());
EditorApplication.delayCall += () =>
{
var player = new Player
{
Name = "Bob",
Health = 100,
Position = new int2(10, 20),
Inventory = new[]
{
new Item {Name = "Sword", Type = ItemType.Weapon},
new Item {Name = "Shield", Type = ItemType.Armor},
new Item {Name = "Health Potion", Type = ItemType.Consumable}
}
};
var json = JsonSerialization.ToJson(player);
Debug.Log(json);
var player2 = JsonSerialization.FromJson<Player>(json);
};
}
public class Adapter :
IJsonAdapter<Player>
, IJsonAdapter<int2>
, IJsonAdapter<Item>
{
public void Serialize(in JsonSerializationContext<int2> context, int2 value)
{
// Serializes as a value as a custom string.
context.Writer.WriteValue($"{value.x}, {value.y}");
}
public int2 Deserialize(in JsonDeserializationContext<int2> context)
{
// Since we serialized a custom string for the value, we can convert the serialized value to a string
// and extract the 'int2' manually.
// This could be optimized to avoid allocations on the string operations.
var stringValue = context.SerializedValue.AsStringView().ToString();
var values = stringValue.Split(",");
if (values.Length != 2)
{
throw new InvalidJsonException("Expected two values to deserialize a 'int2' type");
}
return new int2(float.Parse(values[0]), float.Parse(values[1]));
}
public void Serialize(in JsonSerializationContext<Item> context, Item value)
{
// Serializes as a value as a custom string.
context.Writer.WriteValue($"{value.Name}-{value.Type}");
}
public Item Deserialize(in JsonDeserializationContext<Item> context)
{
// Since we serialized a custom string for the value, we can convert the serialized value to a string
// and extract the 'Item' manually.
// This could be optimized to avoid allocations on the string operations.
var stringValue = context.SerializedValue.AsStringView().ToString();
var values = stringValue.Split("-");
if (values.Length != 2)
{
throw new InvalidJsonException("Expected two values to deserialize a 'Item' type");
}
return new Item
{
Name = values[0],
Type = Enum.Parse<ItemType>(values[1])
};
}
public void Serialize(in JsonSerializationContext<Player> context, Player value)
{
using var objectScope = context.Writer.WriteObjectScope();
// Most primitives can write key-value pairs directly.
context.Writer.WriteKeyValue("N", value.Name);
context.Writer.WriteKeyValue("H", value.Health);
// Sub-objects can use the `SerializeValue` method.
context.Writer.WriteKey("P");
context.SerializeValue(value.Position);
// Arrays can use the `SerializeValue` method.
context.Writer.WriteKey("I");
context.SerializeValue(value.Inventory);
}
public Player Deserialize(in JsonDeserializationContext<Player> context)
{
// Here, since we have manually written all the fields/properties using alternative names,
// we can simply deserialize each field/property using that name and type.
var player = new Player
{
Name = context.DeserializeValue<string>(context.SerializedValue["N"]),
Health = context.DeserializeValue<int>(context.SerializedValue["H"]),
Position = context.DeserializeValue<int2>(context.SerializedValue["P"]),
Inventory = context.DeserializeValue<Item[]>(context.SerializedValue["I"])
};
return player;
}
}
}
Simply put, when deserializing, you can access the serialized data view through the context.SerializedValue property. This serialized data view can either be converted to a known type using the As[...] methods or you can navigate through nested fields/properties by using the [ ] operator or the GetValue method.
For a given SerializedValueView, if you know the type, you can use context.deserialized<T>(view) to deserialize the value. I believe primitives needs to go through the As[...] methods.
Unless I misunderstood what you are asking, I believe there is an example in the snippet I provided.
Seems like a valid bug. I’ll take a note. Again, a fix might take a bit of time while we figure out what will happen with the package now that the owner has left Unity.
The reason we introduced UnsafeAppendBuffer was because we needed something that was burst compatible. Most of the serialization package was written with Burst in mind.
A clone operation is something that is often requested. It is however not something we’re likely to add to the API. There are several reasons for this, but it mostly boils down to what should the cloning operation even do? Should it do a shallow clone preserving all references or a deep clone?
Over the years, we’ve implemented multiple version of the clone operation and in each version, it only served for specific contexts and would be completely broken in others. I think it’s for similar reasons that the ICloneable interface is not recommended to be used in public APIs.
In most cases, it’s probably easier to write custom-tailored visitors for what you want to do.
As far as I’m aware, it should be safe…as long as no calls to Unity is made and no Unity object are created or assigned.
Using a ExceptionDispatchInfo is a very good suggestion.
I’m not very familiar with the validation code in this package, if we do have access to the context during validation, it would be a good suggestion to output it.
I think this should be possible today. If I recall correctly, the reason that we used a boxing interface in this case was because at the time, IL2CPP didn’t support full generic sharing and it required a lot of AOT helpers to be generated.
Changing this would be a breaking change however and would require another major version to be released.
This should already be possible using the current adapters. The adapters are done in a way that they can be used in both contexts. To override an instance, you can use context.GetInstance(), modify it and return it. It’s limited, but it’s a start.
This is true at the moment. The adapters do have access to the visitor, which contains the SerializedReferences, but it is not exposed. With additional validation added to SerializedReferences, it could be exposed.
I think it’s the other way around. What probably happened is that some internal feature required the binary one, so it was made public and we forgot to toggle the JSON one.
I am interested in the Clone because, when writing custom tools or dealing with data not inherit Unity Object, a method similar to Object.Instantiate is needed (Usually, shallow cloning can be achieved through MemberwiseClone). Additionally also Copy is useful, which can be used to reset state, which is useful when using object pools.
Additionally, I am curious about a question. Will Unity Properties & Serialization be used to improve the serialization and instantiation of GameObjects? Is it faster than the existing solutions? If you have conducted performance tests, developers would be very interested in the details.
Object.Instantiate is a good example of how a Clone operation is more complicated than it can seem:
The parent property of a cloned game object will not be set by default.
Prefab connection is not kept (in order to keep it, you must clone the game object by using PrefabUtility.InstantiatePrefab.
The components of the game object are cloned, which requires a distinct way of creating/adding them (they are stored in a different storage and you can’t simply call new MyComponent() and insert them in a list on the game object.
The children of the game object are also cloned.
References to other game objects/components will either be copied in a shallow way (copy the reference if it is outside of the cloned objects) or remapped (if the reference is inside of the cloned objects).
Instance ids are obviously not copied.
If you clone a MonoBehaviour using Object.Instantiate, you end up cloning all the game object, all of its components and all of its children.
This is mostly on top of my head and I’m probably missing a few things. Now this operation is possible because Object.Instantiate knows about all of this. I haven’t checked the native code too much lately, but it probably goes through some of the serialization paths as well.
When using the Entities package, trying to clone entities would be done in a completely different way, even if in essence, the relationship between Entities and their components is similar to the relationship of Game Objects and their components.
While trying to create a generic Clone operation, we concluded that it required a fully customizable system where you could inject global, local and contextual rules. In the end, every single time, we ended up writing dedicated code.
This is very unlikely, as they serve two different purposes and their “domain” so to speak is different as well.
Properties in itself is not tied to serialization at all. It uses the same rules to automagically create properties from fields, but that was done mostly for “backwards compatibility” and avoid needing to instrument existing type too much. I’m still on the fence on that. I think if I had to start over today, I would probably require explicit tags on fields/properties and completely decouple it from the serialization system.
Properties also only has access to the managed side of things, whereas Unity has a lot of native components and data with light wrappers in managed.
Serialization was made for runtime use-cases and to serialize types that Unity can’t. In terms of performance, if we were to serialize scene data using Properties & Serialization instead of the classic Unity serialization, I would wager that it would be slower. It’s a question of exposed features and trade-off:
Properties allows you to fully use C# properties instead of only fields/auto-properties. This means that the serialization/deserialization can go through arbitrary code. This indirection alone would make it slower.
Properties must create property bags for every type that it needs to use. Even if we were to use a source generator to codegen all the property bags at compilation time (which would increase the compilation time), there is a Mono.Jit cost to registering a property bag and visiting a type the first time. Unity can do a lot of the heavy lifting on the native side and avoid a lot of just-in-time compilation.
Properties & Serialization were created to be user facing, extensible features, including the ability to manually write your property bags, create properties that are not to be serialized, support read-only fields and properties, support migration, support adapters, etc. Extensibility usually has a cost. Unity serialization is a much more closed system with few extensibility points.
Properties & Serialization are managed-only libraries. Porting Unity to CoreCLR will help a ton here, but generally, it will probably still be slower than native code.
Properties & Serialization will treat every reference type as a reference, whereas Unity serialization will treat most types as a value type unless tagged with the [SerializeReference] attribute.
But, as I said above, the aim of Properties & Serialization is different than Unity serialization. One aims to enable light-weight features (in comparison) that work in all contexts (editor, runtime, build) and the other is a low-level framework construct that powers a LOT of Unity features (inspector framework, undo-redo, prefabs, presets, etc.)
In working with Properties, I noticed that the Accept methods of Property and PropertyBag accept visitor types as interfaces, it means that value-type visitors will be boxed. However, I need to use value-type visitors because each visitor has independent state.
I could resolve this issue using an object pool or by using a stack/scope to cache the state of the previous visitor, but I wonder why not make Accept generic? Like this:
void Accept<TVisitor>(ref TVisitor visitor) where TVisitor : IVisitor;
We did experiment with making everything go through generics (including the properties themselves) to be able to have unmanaged visitation. The issue, at the time, was that the support for open generic types in IL2CPP was limited and required writing a lot of ahead-of-time helper boilerplate to generate the proper combinations of type. While a lot of this boilerplate could be done on our side, deviating from the most simple use-case (i.e. calling a visitor from a visitor) would require users to write their own ahead-of-time helpers.
The support has been greatly enhanced since then, so it might be possible to do today, but it would be a big breaking change.
As you’re also answering questions related to Unity.Properties, here is one (maybe we should open a new thread about those)
Any KeyValueCollectionPropertyBag, returns its contents as KeyValuePairProperty’s, which makes perfect sense. However, as the “.Key” and “.Value” properties for a KeyValuePair are ReadOnly (which also makes sense, as thats what they are), any ValueType Key &/o Value indexed on a IDictionary will be unwrittable, and cannot be edited through the standard Runtime Bindings as PropertyContainer.SetValue (concreteley the PathVisitor implementation) wont be able to write back any modified value.
I stumbled upon this while writing Editor tools and UI fields with the runtime bindings system and I am currently bypassing it by overriding UpdateSource<T>(...) on a custom DataBinding class, but im hoping to see if by any chance, you have a recommended approach for dealing with this, or if KeyValueCollectionPropertyBag could eventually return a special cased Property that returns a KeyValuePairProperty which modifies the collection when modifying the Key/Value sub-properties, instead of returning a ReadOnly version for each one.
The “Key” property might be problematic, as it would require removing/adding and that probably arises some issues. But i believe the “Value” one should be safe to be implemented this way?