Extreme build performance regression with com.android.tools.build:gradle >= 3.6.x

Unity Engine
, , ,
1

I have had to upgrade com.android.tools.build:gradle, in order to build AABs (which needs 4.0.x) and to support ARCore (which needs 3.6.x) and noticed that certain gradle build steps are nearly 40 times slower than with 3.4.x.

I tracked the regression to the jump from 3.4.x to 3.6.x and it is the same on every version after that.

Here are some Gradle Build Scans for the same build done with different versions of com.android.tools.build:gradle, both done after cleaning the project.

3.4.3 - Total build time of 14.783s
3.6.4 - Total build time of 6m 49.483s (!!!)

The offending steps are:
:launcher:mergeDebugJavaResource 6m 41.434s com.android.build.gradle.internal.tasks.MergeJavaResourceTask
:launcher:desugarDebugFileDependencies 6m 36.926s com.android.build.gradle.internal.tasks.DexFileDependenciesTask
:launcher:mergeDebugNativeLibs 6m 35.912s com.android.build.gradle.internal.tasks.MergeNativeLibsTask
:launcher:processDebugResources 6m 34.827s com.android.build.gradle.internal.res.LinkApplicationAndroidResourcesTask

Any idea what I could possibly do to solve or at least improve this? It massively increases my iteration times.

2

I’ve done a little bit of hunting to see what on earth is being passed as an argument to aapt2.exe to take nearly 7 minutes to complete. To my surprise, I found that it passes the longest regular expression I have ever seen in my life as an parameter to --no-compress-regex.

The regex appears to be a character by character match of every asset in the build, which in our case is a lot of assets. The regex is 189,189 characters long.

Heavily truncated example:

--no-compress-regex (((a|A)(a|A)/(a|A)(n|N)(d|D)(r|R)(o|O)(i|I)(d|D)/(b|B)(a|A)(s|S)(e|E)(a|A)(n|N)(i|I)(m|M)(a|A)(t|T)(i|I)(o|O)(n|N)(s|S)_(a|A)(s|S)(s|S)(e|E)(t|T)(s|S)_(a|A)(l|L)(l|L)\.(b|B)(u|U)(n|N)(d|D)(l|L)(e|E)$)|((a|A)(a|A)/(a|A)(n|N)(d|D)(r|R)(o|O)(i|I)(d|D)/(c|C)(u|U)(r|R)(r|R)(e|E)(n|N)(c|C)(y|Y)(i|I)(c|C)(o|O)(n|N)(s|S)_(a|A)(s|S)(s|S)(e|E)(t|T)(s|S)_(a|A)(l|L)(l|L)_(d|D)0(f|F)18(e|E)(e|E)

What on earth is going on!?

3

Oh, that’s concerning, I assume, it auto converted things assigned to aaptOptions.noCompress in \unityLibrary\build.gradle, do you have a lot of files there?

4

In this case it’s launcher/build.gradle, but that is indeed the issue, that is precisely what it is doing. It would appear that every asset bundle in streaming assets is ending up in aaptOptions.noCompress and the build tools explode that to that monster regex. Replacing that whole list with ‘.bundle’ brings the build time down to nothing again. This leads to two obvious questions: Why was it fast before 3.6.x and why is Unity putting every file in there individually as opposed to just .bundle.

Edit: Actually, dumping the aapt2 args that 3.4.x was spitting out, I can see why it used to be fast, it used to send each one to an individual -0 argument. 3.6.x compiles a monster regex out of the whole thing itself.

Edit 2: Found the offending entry in the changelog:

  • aaptOptions.noCompress is no longer case sensitive on all platforms (for both APK and bundles) and respects paths that use uppercase characters.

That is an… interesting… way of implementing that.

5

I think -no-compress-regex is a new option in android gradle plugin, it’s surprising, that google under hood started using it implicitly. The full paths are specified, because not to collide with other Unity resources.

Could you report a bug with a repro project - https://unity3d.com/unity/qa/bug-reporting, so we can have this on our radar? Thank you

6

Sure, will do on Monday.

7

Hi all!

Recently I did some changes related to how Unity creates no-compress list. If a file has a distinct extension (not shared between compressible and not-compressible assets) then no-compress list will only contain an extension of that asset. This dramatically reduces size of the list in cases where projects contains a lot of streamable assets with the same extension. For those who has the slowdown issue with a gradle plugin I suggest to try it out. It might help a lot. It won’t make the plugin faster, but it will reduce a number of operations it has to do.

The feature is available in:
2019.4 - planed to be backported
2020.3.13f1
2021.1.9f1
and up…

8

Thanks for the heads up. I tried to make a repro against 2021.1.21 and the behaviour has been quite different, so this would explain that. I haven’t tested with my actual project (it doesn’t build in 2021) but I have copied the directory structure of my StreamingAssets across and it does condense everything to .bundle, which is my current workaround on 2019.4.31 anyway, bringing build times from 7 minutes down to the 15 seconds it was on com.android.tools.build:gradle <= 3.4.x. It is obvious that aapt2 was ending up spending literal minutes evaluating that regex.

One other thing that is interesting is that if I force 2021.1.21 to generate an asset list as before (by seeding with files that don’t have an extension) the issue is slightly different than it was on the project generated by 2019.4.31, instead of the build getting ridiculously slow there is a threshold of files past which aapt2 just immediately crashes.

9

Well there was a file limit with older gradle plugins as well. I believe it was around 620 files or so. If more entries would be added to no-compress-options aapt2 would just refuse to deal with it. That’s why I was implementing this workaround I mentioned in the first place.

Just to be clear: You are saying that newer versions of gradle plugin still has this issue?

10

-ish. From 3.6.0 onwards, the Gradle plugin generates a single monstrous regex for everything provided to aaptOptions.noCompress and passes it into -no-compress-regex rather than a series of -0 entries. So, when building from 2019.4.31 (which is missing your fix) it ends up generating an absolutely ridiculous regex for the file list and slows the build down to a crawl, even with fewer elements than 620. When trying to repro, I did get to a number where it was just outright crashing rather than being slow, which sounds similar to the original issue. Either way, your fix should sort both issues out.

11

Thank you for all the insight into this. I knew they added regex but did not test how it performs myself. I’ll try to push backport of the fix to 2019.4 ASAP :slight_smile: