UCB Team: Cloud Build errors December 26-27

Hello - since late yesterday (US Central time) a number of Cloud Builds have been failing with an internal error, that managed to escape internal monitoring. (The most common is an error that is usually seen as a sporadic one-off setup problem on a build system, that’s now recurring.) The team are tracking this down now and will post updates.

The issue has been tracked down and - we think - mostly resolved.

As background: there are various external dependencies that are set up on build systems, and one of them (a ruby component used by the Cloud Build build-running tool) started consistently being unavailable. This meant that an error that has been sporadic in the past (ruby errors due to gems/components failing to install as one-off problems) suddenly started blocking builds in general.

A new version of the ruby build tool has been deployed to production now which removes this dependency, and the team are running test builds - successfully so far. The team here will be revisiting some of the other external dependencies that Cloud Build relies on in the build system processes, as well as updating monitoring to detect when a component installation fails in a way like this (that blocks builds repeatedly). More notes to come, but we wanted to get a quick summary out here now.

@PhilMcJ Has this been resolved? Were seeing a bunch of random errors on some of our builds as well:

25446: [Unity] Player export failed. Reason: Shader error in ‘Sprites/Default’: Internal error communicating with the shader compiler process

@sghamaty This is a separate issue. It is listed in the Known Issues thread and there have been previous posts about it :slight_smile:

Got it, thanks for the link!

This still an issue or tracked? Recent android UCB.

/BUILD_PATH/.rvm/gems/ruby-2.1.1/gems/bvr-1.5.19/lib/bvr/project.rb:83:in `block in install’: failed to update buildpack checkout (RuntimeError)
ERROR: failed to update buildpacks!
ERROR: pre-build step failed to setup environment

@Erothez Please make a new thread for this issue and we can look into this :slight_smile:

  • Sophia