Deserializing GlobalObjectID issue

[zkrizo]

What’s the proper way to deserialize blackboard variables of custom monobehavior based scripts that are instantiated? I’m getting the following error when attempting to deserialize an object:
InvalidOperationException: An error occured while deserializing asset reference GUID=[7a89880255e2246de83870fb9c1e9803]. Asset is not yet loaded and will result in a null reference.

The documentation made it seem like the IUnityObjectResolver would be called on the agent and all of the blackboard variables to be able to translate things between serializing and deserializing but that appears to not be the case. It seems to only be calling the Map function for the Agent GameObject itself and the rest of the data is serialized using the GlobalObjectId which changes between instantiations and causes that error to be thrown.

It seems like the serialization package’s solution to this is to make custom Adapters to handle the specific types and how it serializes and deserializes. Unfortunately, the built-in behavior JsonBehaviorSerializer does not seem to expose any way to add or modify what Adapters the serialization route is taking. There’s also no way to modify the JsonPackageSerializationParameters (where these adapters are set, as well as additional useful settings like being able to minify the serialized json output to save space in the string)

Is the intention for users to make their own entire JsonBehaviorSerializer class, copying the adapters (because they’re private) in the behavior library and expanding with adapters for their own custom scripts?

I found this previous discussion from October (Behavior: Errors deserializing BehaviorGraph JSON after re-creating related GameObjects) where the end result seems to be a workaround that you unassign all blackboard variables before serialization and restore them afterward. With the way I currently have some of my behavior tree set up (Restart/Abort nodes looking for variable changes) that is not a feasible option for my use case as that would alter the agent’s behavior whenever a save happened.

I’ve got a 100% repeatable project I can file a bug with that’s based off of the serialization example but wanted to see if I’m just going about this wrong or there’s some way to fix the error that I’ve not come across

[Darren_Kelly]

Hi @zkrizo,

I am currently working on runtime serialization in general at the moment.
It would be great if you could submit the project & mention the bug so that I can see which way you have implemented things & why you may be seeing that error.

I did fix a similar error in a user’s project some time ago but it may be different in your case, also it will be value for our learnings also so we can solve this for others in the future so they don’t reach the same problem.

[zkrizo]

Hi @Darren_Kelly,

I just submitted it up, should be IN-91405! Let me know if you need more info

[Darren_Kelly]

Thank you! We are currently prepping for a release this week & I have one task to look at first. I can hopefully take a look tomorrow or Friday.

I am going to see if anyone on the team has time also before me.

[adrian-miasik]

Watching this thread very closely

It’s been a minute since I touched the behavior package since running into these issues and I haven’t circled back just yet. Been busy in other domains, but about this:

@zkrizo

I found this previous discussion from October (Behavior: Errors deserializing BehaviorGraph JSON after re-creating related GameObjects) where the end result seems to be a workaround that you unassign all blackboard variables before serialization and restore them afterward. With the way I currently have some of my behavior tree set up (Restart/Abort nodes looking for variable changes) that is not a feasible option for my use case as that would alter the agent’s behavior whenever a save happened.

Would it be possible to temporarily disable the BehaviorGraphAgent to not invoke the Restart and Abort nodes, make property changes, then enable the agent again? Or are these nodes listening at all times?

[Darren_Kelly]

Hi @zkrizo & @adrian-miasik,

I can see the issue now more clearly with what you have provided.
This is actually quite similar to what you reported before @adrian-miasik just in a different way which caused me to find some issues with our current implementation.

I will now look into updating our sample to be more complete & show being able to serialize / deserialize between playthrough’s & add tests to catch these problems in the future.

EDIT: The problem lies with the GameObject resolver not being triggered for your custom type. For now you can use GameObjects instead like I suggested below.

What I did find though is that this seems to be a problem even when assigning a test point object directly inside the Agent prefab which I did not expect. So I could reproduce the error without clicking the add variable button in your provided project which is definitely a bug on our side.

I will try to see if we can find a way solve this in the future somehow via the global object resolver or something similar. Hopefully with this future solution we can then add / remove GameObject’s at runtime via instantiated prefabs and resolve finding them via name or a custom solution from the user.

One workaround for now, could be to not save this object’s reference in a blackboard variable & to assign & remove each time you run.
So before saving, remove the reference. After loading add the blackboard variable reference to the instantiated object again.

I know this is not ideal, sorry for the inconvenience.

I will be looking into this asap with help from the team.
Due to Christmas holidays and lot’s of us being off during this time, this may take some time for a fix to be found and released.

Thank you for your patience & I hope you both have a great Christmas!

[Darren_Kelly]

@zkrizo @adrian-miasik Interesting! I may have found something!

So from what I can see is your custom class is inheriting from Monobehavior so it should be picked up by the GameObjectResolver implementation, but it is not for some reason which is why it’s causing problems with the GlobalId.

That’s great news as it seems to be the root cause of the problem.

For a workaround you can avoid this for now by not using a custom type in the graph & instead using a GameObject. Then inside your node class you can use .getComponent() to get your object type that you need.

I tested in your example and using the GameObject variable instead & allows saving & loading between sessions.

You can add a log in the GameObjectResolver shown in my screenshot to test this.

[Darren_Kelly]

Hi @zkrizo & @adrian-miasik,

I have some good, news! I have found a fix.

This won’t be released until my bigger changes with runtime serialization go out too at the same time though, I am fixing another issue with serialzation between assemblies not working always when inheritance is used between both code assemblies.

In the meantime, you could create your own IBehaviorSerializer and copy the contents of our public class JsonBehaviorSerializer : IBehaviorSerializer
& add this IContravariantJsonAdapter for MonoBehaviours inside the JsonBehaviorSerializer class.

You can see other examples in that class how it’s done similar to GameObjects & Components although in this case we needed a IContravariantJsonAdapter and not just an IJsonAdapter.

Hope some of this info can help you guys get unstuck for now

[zkrizo]

Hi @Darren_Kelly,

Sorry for the late reply, holidays and such kept me busy. I’m glad you’ve found a solution! I’ve tested some things out trying to get to a suitable solution.

For others’ awareness, Behavior’s embedded version of the Serialization package cannot be used to write your own variant of the IBehaviorSerializer as most of the reliant code (IJsonAdapter, IContravariantJsonAdapter, etc.) are locked behind the internal keyword. You will have to pull in the proper Serialization package alongside it (I pulled in 3.1.2). Once you do that, you can write your own IBehaviorSerializer implementation.

I did worry that the Monobehaviour adapter solution would force all Monobehaviour based scripts to be serialized as Monobehaviours rather than a more specific type that can be easily type-checked against. That would lead to a large piece of resolver code trying to figure out what is responsible for that mapping identifier.

For instance:
AIManager for storing monobehavior EnemyController components
PlayerManager for storing monobehavior PlayerController components
LevelManager for storing monobehavior InteractionPoint components
etc.

If each of those were to go into the resolver and just be stored as “MonoBehavior” rather than their true types, the resolver would have to ping each manager and ask each one “Do you have mapping X?” rather than having the true type stored alongside it and the resolver getting Resolve(value) so it would know that it needs to only ask the AIManager to resolve it. Does that make sense?

So, I made additional adapters specific to the derived monobehaviours (IE, TestPointAdapter : IContravariantJsonAdapter in the project I sent up). This works well for serializing and they do come into the mapper with the correct type. On Deserializing, it seems to default to the Monobehavior adapter though which confirms my fears about the above. However, if you instead make your own adapters for your specific Monobehaviour based classes and NOT the Monobehaviour adapter, then it does work properly for both serialization and deserialization. It seems this is due to a priority to the adapters in the list since I added the Monobehaviour one before my specialized ones. Moving the Monobehaviour adapter below the specialized ones in the adapters list does seem to work though I’d probably recommend sticking to the specific types at that point and foregoing the generalized Monobehaviour one.

TLDR (that should probably be added to documentation at some point):

• If you want to serialize custom Monobehaviour based classes in BehaviorVariables you will need to pull in the official Serialization Package. (I had to add it manually to the manifest.json as PackageManager didn’t seem to pull it up)
• You will need to write your own IBehaviorSerializer implementing class
• You will need to write your own IUnityObjectResolver, mapping each specific type
• For scripts that you are mapping yourself with a custom identifier, you will need an IJsonAdapter implementer for each type to call into your IUnityObjectResolver
• All adapters need to be added to the JsonSerializationParameters and passed into the Json ToJson/FromJsonOverride calls

I do have one other issue after all of this has been worked out that doesn’t seem to be working properly. In my main behavior graph, I’ve got loaded agents getting stuck as part of a Try In Order node. They’re loading in in the middle of a downstream sequence which after loading resumes perfectly and the agents finish the sequence to success but the Try In Order node is stuck Waiting and doesn’t seem to be running its OnUpdate method (no logs appear from OnUpdate when adding them) so it just gets stuck. No visible errors or anything

Debug behavior graph:

As for your suggestion @adrian-miasik, the VariableValueChangedCondition subscribes a listener in OnStart and only removes it in OnEnd. I don’t think disabling the agent calls OnEnd so I imagine that disabling and re-enabling would not prevent the event from getting picked up unfortunately. If a workaround hadn’t been found, I was going to start converting things to storing just the identifiers as strings in the BlackboardVariables and having the individual actions/conditions know how to fetch the real instance that identifier pointed to.

[Darren_Kelly]

Hi @zkrizo,

No worries at all, I have been on holiday also!

I am not following the implementation you have completely with what you mention. But I think I understand most parts.

When you say you don’t want the Monobehaviour adapter to catch all types & instead write a IContravariant adapter for each individual.

I guess this is due to the Resolver returning the MonoBehaviour type here, is that correct?

You are correct with the ordering also, the list is prioritized top to bottom.

For your last issue, could you place a breakpoint here in the RuntimeSerializationUtility.cs class & tell me if the JSON is containing the IsRunning property & the ID for the node.

There is an ongoing issue that I am solving and I suspect that the these properties are not serializing correctly for you right now.

[zkrizo]

Hi @Darren_Kelly,

Yes, the resolver passing in the ISerializedType as Monobehaviour was what I was referring to. Great for getting around the GlobalId issue but it will lead to a messy resolver that has to try to translate that monobehaviour back into what it should be rather than an adapter that saves out the true type. That’s why I was recommending to write your own IContravariant adapters for specific monobehaviour based types instead of using the generic one (or at least putting those higher in the priority list than the generic one). Not something you can put something into the package to fix I think, more just knowledge distribution for the documentation.

As for the issue I’m still having with TryInOrder getting stuck, I had two theories that I tried to go through to find if they were the cause.

The first was that implicit sequence nodes were causing issues. By that, I mean if you connect a ConditionalGuard action straight to another node, it seems to create an implicit sequence in the data that gets serialized. I had a thought that the deserializing wasn’t properly handling that implicit sequence so it would break. I went through and attempted to get rid of all of the implicit sequences I’d put in by building the tree in that fashion, converting all of those ConditionalGuard → Node pieces into proper explicit Sequence with ConditionalGuard and Node as the Sequence children. However, the problem still remained.

Theory 2 was something that I noticed when closely following through the data that got serialized. The part that is getting stuck is a TryInOrder root with a Sequence → Sequence → Repeat node with conditions set up in it. I noticed in the data that for some reason, that TryInOrder node had the children saved in a different order (Sequence, Repeat, Sequence) so I modified the Sample Serialization example again to try to recreate it simply. That seemed to work perfectly fine and I couldn’t get it to get stuck after loading. I deleted the connections in my main project and replaced them and now its at least saving the children in the proper order…but it still gets stuck.

The data seems valid, each Node has a $id that seems to be a valid number, their $refs point to other $ids that appear valid, the ID field is filled out but I can’t verify if that matches as none of those IDs seem to be in the .asset file of the behavior graph nor are they the ID in the NodeDescription so I’m not sure where those are connecting. IsRunning seems to be in the proper state going down the chain. Status on that TryInOrder is 4 or Waiting which seems valid. I’m not sure where else to look for where the issue lies.
Would the RIDs in the behaviorgraph.asset constantly regenerating be causing an issue? That’s a headache I’m running into with source control where they regenerate even if I don’t open the graphs

If there’s anything else I can try that would help, let me know.

[zkrizo]

Good News! @Darren_Kelly,

I have figured out the source of the issue! I’m submitting up a project for you to verify to the bug report but here’s essentially what the cause of the issue is (IN-92876).

As we’ve been talking about, I’m instantiating agent prefabs. Immediately after instantiating them, I’m grabbing the behaviorgraph data and deserializing that into the agents. After EXTENSIVE logging, I noticed that there seemed to be duplicate nodes inside of the ProcessedNodes list. I went hunting for where they were coming from and discovered the source: BehaviorGraph.Start.

If you instantiate an agent and deserialize existing data into their behaviorgraph, this all happens before Start fires. Once BehaviorGraph.Start fires, it will go through and essentially runs the graph again, adding already existing nodes into the ProcessingNodes container without regard as to whether they exist already in that list. This eventually leads it into the stuck state that I was trapped in once it processes through the nodes

I highly recommend that the Start process not run if data has already been deserialized into the graph. Whether that’s a block on if ProcessedNodes has values already in it when StartNode tries to run or whether that’s some kind of initialization boolean, I’ll leave to you guys to sort out the best method. For now, I suppose the workaround is to forcibly wait until the BehaviorGraphAgent’s Start method has been called before doing the deserialization process to ensure the BehaviorGraph.Start doesn’t add duplicate nodes into the ProcessedNodes list.

Hope that helps!

[ShaneeNishry]

Hey @zkrizo ,

Thank you for the report and investigation! I believe I noticed this bug also and have a fix for it already, we’ll double check and confirm!

If you want to confirm on your end, in BehaviorGraphAgent.cs find the Deserialize method:

public void Deserialize<TSerializedFormat>(TSerializedFormat serialized,
            RuntimeSerializationUtility.IBehaviorSerializer<TSerializedFormat> serializer,
            RuntimeSerializationUtility.IUnityObjectResolver<string> resolver)

and add these 2 lines at the end:

m_IsInitialised = true;
m_IsStarted = m_Graph.IsRunning;

The final result should be:

        public void Deserialize<TSerializedFormat>(TSerializedFormat serialized,
            RuntimeSerializationUtility.IBehaviorSerializer<TSerializedFormat> serializer,
            RuntimeSerializationUtility.IUnityObjectResolver<string> resolver)
        {
            m_Graph = ScriptableObject.CreateInstance<BehaviorGraph>();
            serializer.Deserialize(serialized, m_Graph, resolver);
            InitChannelsAndMetadata(applyOverride: false);
            m_Graph.DeserializeGraphModules();
#if UNITY_EDITOR
            OnRuntimeDeserializationEvent?.Invoke();
#endif
            m_IsInitialised = true;
            m_IsStarted = m_Graph.IsRunning;
        }

[zkrizo]

Confirmed that does also fix it in the repro project!

[Darren_Kelly]

@ShaneeNishry

on your end, in BehaviorGraphAgent.cs find the Deserialize method:

Happy to see that helped @zkrizo
Is there anything else blocking you with the serialization at the moment otherwise?

[zkrizo]

Not so far, that was the last thing. Everything seems to be working now with the workarounds until a new package with the fixes comes out. Thanks!

[Darren_Kelly]

That’s great to hear! We are working on runtime serialization in general now as well.