U5.3 iOS 9.3.1 Double memory allocation when loading AssetBundle using ODR

Unity Engine
,
1

First off, a bit about how our system works:

When loading a new scene, we first load an empty scene, then check if the required AssetBundle(s) for the next scene exists locally on disk. If so, we load the required bundle/assets from disk. If not, an On Demand Resources (ODR) request is sent and we wait for the request to complete, and the necessary resources to be downloaded, before we continue on with loading the scene.
This works fine, and have been since we implemented ODR in our app.

Suddenly we found that the app crashed due to running out of memory when loading one of our bigger scenes (AssetBundles total ~350MB). We wondered about why this suddenly started happening, as it worked fine just before. We found out that any iOS device running the most recent version (iOS 9.3.1) would crash, while any device running on a lower version than 9.3 would not (we never got to test 9.3 before 9.3.1 came out).

So we started profiling using Xcode’s Instruments. More specifically the “Allocations” profiler.
The scene we’ve been testing is a scene that requires assets from an AssetBundle with a file size of 107MB.
We tested our app with two specific cases:

#1 - AssetBundles are bundled with the app as it gets installed by Xcode:
A data folder containing all our AssetBundles gets built with the app. This means that since the bundles already exist locally on the device as a scene gets loaded, we do not have to perform an ODR request to download the required assets.
What we found was that as the loading functionality starts, allocated memory is ~65MB. The app then checks for and loads the necessary AssetBundle (107MB). In real time, we see that the allocated memory increments up to ~170MB in size, the scene load finishes, and the scene gameplay starts. Allocated memory still hovering around 170MB.

#2 - AssetBundles are not bundled with the app as it gets installed by Xcode:
This of course means that since the necessary bundles does not exist locally, we perform an ODR request to download them. Same functionality as before, only that we have to wait for the ODR request to finish downloading the assets to the device before we can load them.
What we found with this setup is that as the scene loading starts, allocated memory is once again at ~65MB. The downloading of the assets starts.
We see that, just as before, allocated memory starts incrementing in Instruments. However, this time (with ODR) it doesn’t stop at ~170MB. It keeps incrementing to ~280MB of allocated memory.
At this point, the scene load seems to finish as before, but the allocated memory suddenly drops back to the “normal” amount of ~170MB, and the gameplay starts.

So from these two cases, it seems that when an ODR request/download is required, allocated memory increments to about double the amount of the size of the AssetBundle. As if the required bundle is referenced/allocated twice in memory. But this happens only on devices running iOS 9.3.1.

Summarized, our results are the following:

Device running iOS 9.2.1, without ODR:

  • ~65MB of allocated memory as scene loading starts.
  • Allocated memory increments up to ~170MB.
  • Allocated memory stops at ~170MB
  • Scene gameplay starts.

Device running iOS 9.2.1, with ODR:

  • ~65MB of allocated memory as scene loading starts.
  • ODR download starts.
  • Allocated memory increments up to ~170MB.
  • Allocated memory stops at ~170MB
  • Scene gameplay starts.

Device running iOS 9.3.1, without ODR:

  • ~65MB of allocated memory as scene loading starts.
  • Allocated memory increments up to ~170MB.
  • Allocated memory stops at ~170MB
  • Scene gameplay starts.

Device running iOS 9.3.1, with ODR:

  • ~65MB of allocated memory as scene loading starts.
  • ODR download starts.
  • Allocated memory increments up to ~280MB.
  • Allocated memory drops to ~170MB
  • Scene gameplay starts.

All tests are performed on Unity v5.3.4p6.
We used an iPad Air 2 and an iPad Pro for this particular test. The same crash behavior occurs on all our other devices as well (iPhone 6, iPhone 6s, iPad Mini 2).
This particular test case were carried out on one of our smaller scenes (107MB bundle size).

Just for extra information, the scene that crashes, which I mentioned earlier, requires two AssetBundles with a size of 140MB and 210MB respectively. Which means that if these sizes are in fact doubled on ODR download, it will definitely cause a crash.
.

Have anyone else experienced this or similar behavior after upgrading a device to iOS 9.3.1?
Any ideas as to what may be happening, or any tips as to what kind of workaround we may try to apply to fix this?

We haven’t come across any patch notes or seen any other posts regarding such an issue on either Apple’s or Unity’s sites/forums as of yet, so we’re kind of stumped as to what is actually happening here.

I know this may be just as relevant to post on an Apple forum, which I will as well.

Thank you for reading. Sorry for the long text and anything that may not be explained or worded properly.
Looking forward to hearing your thoughts.

EDIT: We’re using IL2CPP backend, targeting Metal/ARM64. Just ask for any other info.

1 Like
2

Hello,

One bug that caused memory leaks when using ODR has been fixed in 5.3.4p6. Could you please verify that you indeed are using this or newer release? If it’s the case, could you please submit a bug report, attach the reproduction project and post the bug number here? I would like to look into this.

3

Thanks for the reply.

Yeah, we’ve double checked that it’s not related to that specific memory leak. This seems to be something else.
We’re not able to upload our current project in a bug report (way too big), so we’ll have to create a separate project with the same type of functionality.
The only issue with that is that the app + bundles/ODR resources have to be archived and uploaded to TestFlight/iTunes Connect, so we have to create a new project there to “host” the ODR resources or create some kind of way to get around the Apple/iTunes Connect side of things.

I’ll update this thread when we get to that. For now we’ve gotten around it by having the bundles for the scene that crashes installed with the app, though that of course increases the initial install size of the app by ~350MB. And that’s not really a solution to the problem.

As a side note; we just tested today with iOS 9.3.2, and the issue remains (still on Unity 5.3.4p6).

4

A bug report with the exact same issue, and results, as I’ve outlined above was submitted a couple of days ago.
It’s currently sitting on Priority 2 and Severity 2, though I would think such an issue would be higher up on the list.

Direct link to issue:

Issue ID: 798641

Hope that helps you guys with testing and hopefully solving the issue quickly.

EDIT: Thanks to whoever submitted the bug report/repro case. Would have taken us a good amount of extra work and time outside of our already crammed schedule.

5

Any updates on whether this issue has been addressed at all yet? :slight_smile:

6

Hi,

This turns out to be an iOS bug. Unfortunately we could not find a good work around for it.

A hacky work around would be to use regular bundle resources instead of Asset catalogs for release version of the app. They can be assigned ODR tags and act like a regular on demand resource. Then the path of the downloaded resource could be retrieved via NSBundle pathForResource API in a native plugin.

A bug has been filled to Apple, radar# 26715824.

7

Thanks for the update. Good to know, and we hope Apple will be able to resolve it.

As we’re not able to see other people’s bug reports, would you mind posting here again once you hear anything from Apple developers regarding this issue?

We can also submit a report on this issue ourselves and refer to your case number, as it seems that Apple prioritizes and sets severity based on the amount of bug reports submitted on a specific issue.

8

Seconded, since we can’t see the bug report through search, it would be useful to see what you’re running into. Experiencing this as well in a release.

9

Yesterday, we decided to give the iOS 10 beta a shot for one of our devices. What we noticed was that the double memory allocation had gone away!
Now, we haven’t yet had the time to do thorough tests of all cases with different devices etc., but it seems as though this issue may actually have been fixed with the update to iOS 10.

@Shaun-Peoples
Is it possible for you to upgrade to iOS 10 on one of your devices and see if that resolves the issue for you as well?

@povilas
Have you heard anything back from Apple since your filed the bug report about this issue? Have Apple mentioned anything about this issue being fixed or postponed to come with the update to iOS 10?

We’ll continue to do some more testing over the next couple of days. Hope to hear back from you guys soon with any results you might have.

10

Yes, Apple said they fixed the issue in one of the iOS 10 betas.

11

Yes, sorry for not updating this topic.
As mentioned earlier the iOS 10 beta (and now the official version) seems to have fixed the issue. As well, it seemed they fixed it in the latest 9.3 patch as well, so seems all is good now.

Thank you very much for your support @povilas :slight_smile:

12

Oh, never mind. I don’t think it was fixed in iOS 9.3. We just did what you suggested before and created a workaround for it to work on 9.3. I don’t think it’s officially fixed before iOS 10. :slight_smile:

13

I think I have the same issue, but not using ODR, but using Asset Bundle variants (Asset Bundles that are converted to Asset Catalogs when exporting the game for Xcode).

When loading the Asset Bundle from the catalog on iOS 9.3.5, the memory consumption is going up according to the size of the bundle, then drops to something below (but still is higher than on iOS 10). On iOS 10 there’s only a minimal amount of memory consumed when loading the bundle (I guess it’s solely the metadata).

Unfortunately I can’t compare the behaviour on devices pre iOS 9.3, b/c we don’t have any at hand.

@mortennost : How did you workaround the problem?

@povilas : do you have any news on how to solve this issue?

14

Hi,

I am having a little different problem. I am downloading 30+ asset bundles on splash image. When i am downloading it gets instantiated or gets uncompressed in the memory and game gets crashed. I have downloaded the bitbucket AssetBundleManager project. When i am building the asset Bundle i am using chunkedbased compression.

#if ENABLE_IOS_ON_DEMAND_RESOURCES
                if (PlayerSettings.iOS.useOnDemandResources)
                    //options |= BuildAssetBundleOptions.UncompressedAssetBundle;
                    options |= BuildAssetBundleOptions.ChunkBasedCompression;
#endif

I just want to download the asset bundle but don’t want to get loaded in the memory. Basically i am using asset bundle sample script… LoadAssets. But i don’t want to load… i just want to download the bundle and then later i can load it in the memory when needed.

Can someone just point me to the right direction.