(Case 1174117) Addressable dependency resolution is unreliable

Unity Engine
, ,
1

So, we have been stuck on 0.5.3-preview for a long while now, which has been blocking upgrading to Unity 2019, and this is something that I have been trying to tackle. With every release after 0.5.3, our game just does not work. So, I have now rebuilt the Addressables setup and catalogue from scratch in 1.1.7, fixing all asset duplication issues we had at the same time. To my great surprise, the game still does not work.

After much investigation, I have concluded that there is something very wrong with dependency resolution. We have very complicated dependency graph, assets depend on other assets, which depend on other assets, etc. I have made sure that anywhere where an asset depends on an asset in another asset bundle, the latter is addressable so that it is explicitly referenced by that bundle.

There are no duplicated dependencies anywhere at this point, both by design and by analysis rule.

Now, when we load, say, the main character, we get a lot of spam, which did not occur in our old setup in 0.5.3:

The referenced script (Unknown) on this Behaviour is missing!
(Filename: C:\buildslave\unity\build\Runtime/Scripting/ManagedReference/SerializableManagedRef.cpp Line: 197)
The referenced script on this Behaviour (Game Object ā€˜ā€™) is missing!
(Filename: C:\buildslave\unity\build\Runtime/Mono/ManagedMonoBehaviourRef.cpp Line: 333)

According to other threads, this is caused when you have dependencies multiple levels deep with duplication. This is not the case here. Moreover, what follows is even weirder.

The system uses a very bespoke version of UMA. UMA serializes character recipes as text, which can live in assets, and then references to other assets use a soft-reference system that is similar to addressables. We scrapped the latter, when recipes are serialized any references to unmanaged assets are put in an array of UnityEngine.Object so that Unity can detect dependencies and are only stored in asset files. Loading arbitrary text recipes is not supported. At deserialization time, managed references are restored using the managed reference array. This has worked for a long time now without problemsā€¦ until the missing script spam started.

Now, all of a sudden, Iā€™m getting invalid cast exceptions. What is occurring is that the native objects are deserialized correctly but their managed wrappers are instantiated as UnityEngine.Object! If I debug that code, the name of the object is correct, setting Selection.activeObject to it I can see in the inspector that all the data is there, but object.GetType() returns UnityEngine.Object!

What is even worse is that this seems to be entirely non-deterministic. Which objects fail to load varies.

I have managed to make a repro on a test project using nothing but ScriptableObjects and I am utterly stumped as to where to go from here.

A, B, C and D are separate ScriptableObject-derived classes. A, B and C have a UnityEngine.Object reference. I created the following asset structure:
4818707--461783--graphviz (1).png
All four chains (i.e. objects A1, A2, A3 and A4) are referenced by a prefab. The prefab and all objects in all chains are addressable and all in a Packed Separately group.

Chains 3 and 4 always deserialize correctly.

Chains 1 and 2 occasionally deserialize correctly. Most of the time, the reference to object B is dropped to an instance of UnityEngine.Object. Inspecting the new object with SerializedObject/SerializedProperty I can see that the native object is deserialized correctly, but I cannot do anything with it, since it is the wrong class in the managed domain.

If I change the types of the references to be more specific, the references are outright dropped to null instead.

ā€¦ ideas?

Edit: I have now submitted this as a bug. @unity_bill @PaulBurslem it would be great to get your input on this, I am utterly stumped after spending an extraordinary amount of time in the guts of the system trying to debug this, even making asset dependency diagrams to see what on earth is going on.

3 Likes
2

You wouldnā€™t happen to have any ideas, @Ryanc_unity ?

3

Ya, this is due to an engine bug that has a fix currently going through the approval and merge process now. Will let you know when it is fixed and back ported.

1 Like
4

Ah, cool. That makes sense, I did manage to repro it with both the version of addressables that worked for us and the latest, which would mean that the original wasteful packing we had just didnā€™t trigger this. I guess I need to park my task (and hence the 2019.x upgrade for our project) until the fix is released.

1 Like
5

So, QA have reproduced this issue and have also been unable to reproduce it in 2019.3a12, which is the version @unity_bill has already said this fix should land in. Now, this blocks us upgrading to 2019.x, so I hope itā€™ll be backported at least to 2019.2. Itā€™d be good if it was backported to LTS and 2019.1 too, although I suspect that the latter is not going to happen, as the stable TECH stream release is 2019.2 now.

6

Was just about to post in here that the fix landed in 2019.3a12 just the other day. Backports have been opened for 2019.2, and 2018.4. As you mentioned a backport to 2019.1 wonā€™t happen as the tech stream is now 2019.2. Sorry I missed that cutoff window.

2 Likes
7

I can confirm that the issue appears to be fixed. Just updated a fresh clone of the project to 2019.3.0a12 and for the first time since 0.5.x everything works, with our new zero-dupe packing strategy and everything. Fingers crossed for the first non-alpha version with the fix on, the game is live so we cannot have the rest of the team on the alpha.

One issue, the fix did not make it to the changelog. Neither my bug number nor the original issue in the issue tracker are anywhere in it, or anything that sounds like it for that matter. Thatā€™ll make it difficult to track what version the fix lands in on the non-alpha releases.

8

Looks like this didnā€™t make it into 2019.2.1.

9

So I have just been informed that it is going to be at least 2019.2.5 before this fix lands into TECH and 2018.4.9 before it lands into LTS. This is very disappointing, to be perfectly blunt.

It also raises a lot of questions as to the viability of the interrelation of Unity Package releases and core engine releases. We spend time evaluating packages, suggesting improvements and so on, only to end up in a vicious circle of a new Unity release requiring a new version of a package, which triggers an engine bug, which is fixed by a newer version of the engine, which is in alpha, which might at some point, fingers crossed, be back ported to the versions we can actually release with.

This is complicated even further by the fact that this is happening across multiple packages, not just Addressables. We are forced to update to the 19.x cycle because of issues in LWRP that are fixed by newer releases that require that version of the core engine.

3 Likes
10

Beautifully said

11

First a small status update, it looks like the fix is on track for landing in 2019.2.4f1 unless something critical occurs such as a critical security flaw requiring an update. Which happened last time I gave exact dates on a fix on the forums just like this. The LTS release is still looking like 2018.4.9f1 as we require backports to land in the higher versions first to avoid situations where a user updates from latest 2018 to latest 2019 and functionality has reverted just because a fix has not landed.

I realize itā€™s really frustrating waiting for fixes to land, especially since I was the engineer that worked on this particular issue and really try to make it so fixes can be landed in backports as quickly and smoothly as possible. Iā€™m really sorry I could not get this fix out to you all sooner, but we do have a process in place that ensures quality of backporting fixes just so we donā€™t introduce more bugs than we fixed. This process, as well as the process of verifying the release before we ship it out takes time.

4 Likes
12

so given that latest of 2019.2 is .1. We can expect this to land in 2019.2 in about 4-5 weeks?

13

Thank you for the update. Basically, I got an email from QA that it the fix is approved for release but not added to the queue to be backported yet and that with the current release schedule that means ā€œat leastā€ 2019.2.5, which would be quite a while. It was specifically the fact that it looked like it was not even in the queue yet that worried me, as I could see .5 becoming .6, becoming .7 and so on.

To be honest, the main thing that I found frustrating is that the bug was fixed while 2019.2.0 was out. It was understandable that it was not going to hit .1 or .2, there is no way it could have been merged and QAā€™ed on time, but five point releases later with a qualifier of ā€œat leastā€ was a bit much, considering how much time both myself and everyone else who has posted here with this problem have lost on it. .4 is a much more reasonable prospect.

Still, my point stands, there is a big problem with the way package and Unity versions interrelate, especially with preview packages. For example, the way Unity alphas and betas are run at the moment, submitting bugs and then posting on the forum tends to result in rapid resolution (thanks @LeonhardP ! :slight_smile: ). When, however, a preview package is being tested and it unearths issues in the core engine, the same priority is not given. Seeing how packages lose support for older (even LTS) versions of the engine rather quickly, that is not ideal.

14

Tech releases usually are 1 release a week. 19.2.1 was released the 14th so 19.2.4 should be the 4th. Again, this is if everything goes smoothly with 19.2.2 and 19.2.3.

Monday when I was providing that information to QA, it had not been added to the queue, it was just added yesterday. QA was also trying to cover his bases with the possibility of it getting pushed back due to unforeseen release issues this the ā€œat leastā€ note. Also I read the release calendar incorrectly which didnā€™t help the estimate I provided him. Iā€™m fairly certain I read the release calendar correctly this time.

I fully understand that frustration, and that is definitely something I feel we can do better on for communicating expectations of packages and expected version support while it is in preview. For us, Addressables & SBP have very little integrations into the editor, and because of this, we have not had to make breaking changes to require changing the minimum supported Unity version. However other packages do not have as small as an editor footprint which is where situations like what you are describing with the LWRP have come from.

One of the biggest reasons for the slow turn around on this fix was that I almost had to make a breaking editor change that would not have been able to be backported and struggled with finding a way to fix this without needing to break compatibility for close to 3 weeks.

1 Like
15

Thank you for the explanation, and thatā€™s really good, it allows us to transition to 2019.2 a fair amount of time before the next release and the workflow improvements Iā€™ve achieved with the latest addressables, the new catalogue and my performance fixes are tremendous. Bundle building time was reduced from anything between 1:30hr and 4hr down to minutes when a lot of content is cached, which is most of the time.

It would be beneficial to be upfront about this stuff, as we can then be more confident when testing preview packages. The more we can test them, the closer they will be to our needs and expectations as developers upon release.

I do have to admit, itā€™s spoiling me that I can go in, investigate and fix issues in our build process without hitting the black box that is the native side of the engine for people that donā€™t have a source licence quite as often, this was absolutely not the case before the SBP landed.

Understood, I expect that part of the engine to be a bit of a minefield, considering how much would break if it goes wrong.

16

Another small update, it went though the final queue really quickly and is actually on track for the 2019.2.3f2 release. This bumps the 2018.4 estimate up to a possible 2018.4.8f1 release.

Again, this is if everything continues to go smoothly.

2 Likes
17

This is excellent news! Thanks Ryan!

18

So, the fix has landed on 2019.2.3f1! One slight bit of weirdness, getting a lot of missing script warnings that were happening when the error was occurring. These warnings do not happen in 2019.3.0b1.

1 Like
19

Have you rebuild bundles with the landed fix?

20

Yep, cleared the build cache and rebuilt several times since then. The dependencies are now loaded correctly, the only hint of the issue having being there is the warning.

Interestingly enough, the warning does not happen on my repro project, only on the real project, and so far it appears to be harmless.