What you are talking about is more complex and less performant than it needs to be. I recommend trying to work the way Unity’s terrain does, first. That is, the color channels of your blend mask are multiplied by the textures, and the results are added together. The result needs to add up to one, though, and I can’t recommend any tools that do that. Please let me know if you can.
If you can get that result, then you can paint greyscale “light” in the alpha channel of the vertex colors. Ideally, you could have the alpha channel multiplied into the RGB colors, which would save a multiply instruction in the vertex shader. But even better, if you have a better painting tool, you could get four textures blending, not just three, by blending with all four channels. (This would require that the painting tool use more than four channels internally, in order to perform live blending and light calculations.)
There’s a lot of potential for vertex colors, but personally, I can hardly harness their power. I use Blender, which is god-awful for vertex coloring. It doesn’t even handle the alpha channel; I have to do stupid hacks to get that exporting. I know how to deal with them in shaders though, so if you learn of a tool that can help with the stuff I’m describing, again, please let us know, here.
Also, you can’t run this on hardware that won’t run iOS 5 (pre-3GS). There’s not really a way to do a fallback, either, without other assets, so I recommend forgetting about them. If that’s not an option, you’ll have to drop down to only two textures instead of three or four.
Shader "Vert Color Mix (RGB)" {
Properties {
_RedTex ("Red", 2D) = "" {}
_GreenTex ("Green", 2D) = "" {}
_BlueTex ("Blue", 2D) = "" {}
}
SubShader {Pass {
GLSLPROGRAM
varying vec2 uv;
varying lowp vec3 color;
#ifdef VERTEX
void main() {
gl_Position = gl_ModelViewProjectionMatrix * gl_Vertex;
uv = gl_MultiTexCoord0.xy;
color = gl_Color.rgb * gl_Color.a;
}
#endif
#ifdef FRAGMENT
uniform lowp sampler2D _RedTex, _GreenTex, _BlueTex;
void main() {
gl_FragColor = vec4(
mat3(texture2D(_RedTex, uv).rgb, texture2D(_GreenTex, uv).rgb, texture2D(_BlueTex, uv).rgb)
* color, 1
);
}
#endif
ENDGLSL
}}
}