It seems that for some reason when using Camera.SetTargetBuffers(RenderBuffer colorBuffers, RenderBuffer depthBuffer), only the first color buffer gets the proper color from things within a UI mask. Things outside of masks render fine into both color buffers.
Heres how the buffers are set.
theCamera.SetTargetBuffers(new RenderBuffer[] { rt0.colorBuffer, rt1.colorBuffer }, rt0.depthBuffer);
Both buffers are declared like this.
new RenderTexture(Screen.width, Screen.height, 24, RenderTextureFormat.ARGB32);
Heres an example of the results, the shaders write the same color info to both buffers. Top left is the second render buffer with masking enabled, top right is first render buffer with masking enabled, bottom left is second render buffer with masking disabled, bottom right is first render buffer with masking disabled. The difference between the bottom ones is expected due to some elements not having shaders that write to 2 color outputs. The top 2 show that the second color buffer is not getting color from things within side the mask, and that it seems that somehow it is failing the stencil test, even though the writing is happening within the same fragment program.
The frag program in the shaders look like this. The stencil settings are copied out of the default UI shaders.
struct f2s {
fixed4 col0 : COLOR0;
fixed4 col1 : COLOR1;
};
f2s frag(v2f IN)
{
half4 color = tex2D(_MainTex, IN.texcoord) * IN.color;
f2s output;
output.col0 = color;
output.col1 = color;
return output;
}