Cloud Build queue issues and failures - January 12, 2017

Hello -
Cloud Build is seeing a number of builds queue up and/or fail due to an internal error right now. It’s being addressed now, more details to come!

There may also be some duplicate build attempts appearing for some projects right now. This is also being investigated during the maintenance that’s still in progress.

Update: Cloud Build is mostly back to normal.

A quick recap of what just happened: during a code update, a problem appeared which hadn’t surfaced in testing: some build attempts wound up getting queued multiple times, a problem which was magnified by source control polling. This led to lots of redundant build attempts appearing in some project histories unexpectedly, and also most of them consequently failing with strange errors. (It also crowded out the normal correct build attempts.)

The offending code was reverted, and the queue was cleared to remove these redundant build attempts. Some build attempts that should have been started correctly - either from polling or directly - were also cancelled, so if a build attempt that you’re looking for disappeared, you’ll simply need to restart it (or let polling pick it up on the next change to the repository, if auto-build is enabled).

Some builds will show queue times (or “sent to builder”/preparing to start times) of a few extra minutes for a little while longer, and a few more builds than usual may experience “termination of channel” errors - these are all due to the load currently settling out on the Cloud Build system, and should be fully back to normal very shortly.