Concerningly long build times

Build & Deployment Build Automation
,
1

Hello,

We have a mid sized project that we recently tried Unity Build Automation on. It’s a VR project that builds for Android. Our workspace is 9.4 gb, as reported by Unity. Our APK comes out to 700 mb.

On my local machine, incremental builds take 1:58 (one minute, 58 seconds). However, on Unity Build Automation, they take 42 minutes to 44 minutes. One of our goals for build automation is to speed up the process of a developer creating an APK (2 min), uploading it to the cloud (20 minutes), and having another team member download the build (~1 or 2 minutes). Unfortunately, with build times of 40+ minutes, it’s a time-loss for us.

I understand cloud builds are expected to be slower than local builds, but the degree of slowness is concerning (especially when we are paying by the minute).

I went to investigate and found a few strange things:

[2023-04-01T05:21:09.283Z] - 7.3.3.2.4.2 - INFO: ++++++++++++++++++++ SECTION ++++++++++++++++++++
[2023-04-01T05:21:09.283Z] - 7.3.3.2.4.2 - INFO: + Checking out Source Code
[2023-04-01T05:21:09.283Z] - 7.3.3.2.4.2 - INFO: ++++++++++++++++++++ SECTION ++++++++++++++++++++
[2023-04-01T05:21:10.384Z] - 7.3.3.2.4.2 - INFO: BVR: SCM Checkout
[2023-04-01T05:21:10.387Z] - 7.3.3.2.4.2 - INFO: GIT: Checkout project with depth of -1...
[2023-04-01T05:21:10.617Z] - 7.3.3.2.4.2 - INFO: GIT: git version (2.31.1)
[2023-04-01T05:21:10.617Z] - 7.3.3.2.4.2 - INFO: GIT: Checking out git@git.assembla.com:xxxxxxxxxxxx.git/master with depth of -1
[2023-04-01T05:21:10.765Z] - 7.3.3.2.4.2 - INFO: Attempt 1 of 5 to clone repository.
[2023-04-01T05:31:46.456Z] - 7.3.3.2.4.2 - INFO: Cloned in 635 seconds.

Incremental pulls of the project seem to take a full 10 minutes. This project was already checked out and built several times. (the first build took 3.5 hours, so I’m fairly confident it’s not doing a clean checkout here). The use of the word “clone” and “Checkout” muddies the waters here a bit.

As a point of comparison, a “git pull” on my local computer completes in 1.28 seconds.

It also automatically compresses and decompresses the library between builds.
Extracting and restoring the library from the remote cache takes 3 minutes. Compressing the library takes 14 minutes. The APK is built but inaccessible until this step is completed.

[2023-04-01T05:49:06.063Z] - 7.3.3.2.4.2 - INFO: Zipping cache files from Library
[2023-04-01T06:03:08.834Z] - 7.3.3.2.4.2 - INFO: Created the archive file.

The actual build time recorded on the cloud is 5 minutes, which is pretty close to the local 2 minutes… but there’s just an extra 35 minutes of “other stuff” that we get billed for and adds a lot of dead time to our team’s dev cycles. More concurrency isn’t really helpful for us.

Since this extra library compression and decompression takes so much time:
($0.02 * (14 +3)) = $0.34, per build

We’d rather pay the Unity rate of $0.14/GB/month to not spend the time compressing our 22gb Library folder. It would pay for itself in just 9 builds per month.

I’m also not sure what’s going on with the source control step taking so long. We were expecting cloud builds to take 2x or 3x the time of our local builds, not 20x. Is there anything that can be done to improve this process for us?

Thank you!

2

Since these are not long lived machines, each time we run a build we are doing a full clone of the source repo. Better support for shallow cloning is something that we are working on but do not have a current timetable on when that will be complete.

You can disable compression at the end by making an API call to update the target config.

This may result in reduced time spent zipping the Library at the end of a build and potentially faster performance when restoring the cache. Updating the cache compression isn’t available in the UI but can be configured via the Cloud Build API.To enable no compression make a call to the /orgs/{orgid}/projects/{projectid}/buildtargets/{buildtargetid} endpoint with the following JSON payload:
{
“settings”: {
“advanced”: {
“cache”: {
“compressionMethod”: “none”
}
}
}
}

3

Thanks for the reply! We will experiment with the compression disabled and see how it helps. I appreciate that tip!

I had assumed since it was caching the library folder, it would cache the rest of the assets for the project, that’s interesting to know. So that will also download all the files in the repository, including files that have been deleted (but are still in the git history).

We would eventually like to use cloud builds as a form of continuous integration, so that means running after each commit. Most of our commits are very small code changes, so incremental pulls would be a pretty huge time savings for us.

4

I’m seeing the same issue with long Library cache compression times, but apparently only on the Windows builders. We build our game for iOS and Android, using the Mac builder for iOS, and Windows builder for Android.

“Download… (and) Extracting cache files” takes around 30s on the Mac builder, and 3m 30s on the Windows builder.

For a build with no asset changes, “Zipping cache files from Library” takes 2m 30s on the Mac builder, but it takes at least 18m on the Windows builder. Builds with asset changes can take >30m to zip on the Windows builder.

Assuming this is an issue only on the Windows build machines, why would the archive zipping process be so much slower? Like others, we are optimizing our build and repo to reduce build times and build costs, but this doesn’t seem to be something we can address. (outside of paying a smaller cost for an uncompressed cache)

Note that I have not tried building Android on the Mac builder recently, as it will be cost ineffective.

5

If you disable compression are you still seeing performances like this? Unfortunately there will be differences with some of these tasks between systems because the Windows and Mac machines have different specs which can directly impact the performance of compression.

6

I set compressionMethod to “none” on the Android build target that uses the Windows builder, then ran the build target twice, both times with no code or asset changes.

“Download… (and) Extracting cache files” was slightly faster by about 30s. The “Zipping cache files from Library” step dropped from 18m in the previous build (with compression enabled) to under 9m in the latest builds with compression disabled.

So yes, for this test, the library caching step on the Windows builder appears to be ~2X faster with compression disabled. It is still quite a bit slower than what I’m seeing on the Mac builder, so I’m guessing that compression isn’t the only performance impact.

But if these uncompressed cache times hold true, the total metered cost should be about the same as on the Mac builder.

7

Hello! I went through a new phenomenon that I don’t understand. Perhaps because of the bug, I reduced the build time from 7 to 11 hours to 10 minutes.
!!! It probably won’t help those who only build. However, if it suddenly takes a long time after moving a package like me, you can refer to it.!!!

The original project took 10 minutes to build, but the new project took 7 to 11 hours to build after exporting the contents of the project as a package. The project replicated with GitHub takes about 10 to 20 minutes. Only projects created with exported packages are problematic.

I wasted over 20 days thinking that this is a difference in setting between my original project and my new project.
It’s no exaggeration to say that I’ve touched this and that settings to solve this phenomenon and done almost everything.
I always had a problem with the compiling shader part, so I thought the problem was my URP shader or custom shader problem, of course. (See the image below… Where it’s always stopped is the compiling shader part in any setting.)

But today, I tried to find the culprit who split the package into pieces with my colleague’s idea to make it take a long time to build, and as a result, the build was completed in 7 minutes. There was no culprit in the package.
Below is how I worked.

  1. Export the entire package from the original project. (OR Export multiple packages.)
  2. Import only a portion of the package exported to a new project.
  3. Build the project with only a portion of it imported.
  4. Try 2) to 3) until you move all of them.
  5. Build after moving everything! Done.

!!! It probably won’t help those who only build. However, if it suddenly takes a long time after moving a package like me, you can refer to it.!!!

I thought it was a compiling bottleneck, so I did a clean build, but the results were the same.

I’m using a translator, so I’m sorry if the content is weird! I Bless your code!