Quick correction. There are no “hit” vertices. The projector is doing an intersection test of the projector’s frustum and the bounds of meshes in the scene and rendering any mesh it overlaps with an additional time with the projector’s material. The vertices are transformed into the camera’s clip space and into the projector’s screen space. Clip space (aka projection space), is a little different than screen space, and you’ll have to understand that a bit. Applying a camera’s projection matrix to an position that is in view space will put it into clip space.
That UnityObjectToClipPos function applies the full MVP (Model View Projection) matrix, which transforms a point from the local object space mesh position to the clip position, hence the name. This diagram may help:
source: https://twitter.com/capnramses/status/936393671519494151
Note the names of each “space” isn’t rigidly defined, so they’re called different things by different people or in different places. Even within Unity’s shader code there’s not a ton of consistency. However it’s good to note that the names of the matrices (on the arrow lines) in the above diagram match up well with Unity’s usage of “Model”, “View”, and “Projection”. An MVP matrix is the model, view, and projection matrix multiplied together so that transforming from local or object space into clip space can be done in a single matrix and vector multiplication.
edit: If you’re curious about the “normalized device space” and “viewport space” in the above diagram and why that doesn’t seem to happen in the shader, it’s because the GPU does that on it’s own. The GPU expects a clip space position output from the vertex shader and does the transformations into viewport space as part of rasterization.
The unity_Projector is essentially a MVP matrix for the projector, but setup so a position is transformed into a “UV” space (essentially 0.0 to 1.0 range “screen space” for the projector).
The unity_ProjectorClip matrix appears to be a similar matrix, but is always an orthographic projection. The reason for this is the depth in clip / projection space is not linear (0.5 is not half way between the near and far clip planes in world space), but for orthographic projections it is linear in world space. You probably won’t need this.
So what you need is the “MVP” for each of your camera views so you can get it into clip space. However it’ll be a lot easier if you just pass the VP matrix so you don’t have to worry about the mesh’s local to world matrix. You can then transform that into “UV” space fairly easily in the shader using the same way Unity does to get screen space positions rather than trying to figure out how to get the matrix setup properly.
psuedo shader code:
// vertex shader
float4 worldPos = mul(unity_ObjectToWorld, v.vertex);
float4 camA_ClipPos = mul(_CameraA_MATRIX_VP, worldPos);
float4 camB_ClipPos = mul(_CameraB_MATRIX_VP, worldPos);
o.camAPos = float4(camA_ClipPos.xy * 0.5 + camA_ClipPos.w * 0.5, camA_ClipPos.zw); // magic!
o.camBPos = float4(camB_ClipPos.xy * 0.5 + camB_ClipPos.w * 0.5, camB_ClipPos.zw);
// fragment shader
float2 camAUV = i.camAPos.xy / i.camAPos.w;
float2 camBUV = i.camBPos.xy / i.camBPos.w;
In C# you can calculate the MATRIX_VP by doing this (assuming you setup dummy cameras objects to align to the camera views instead of projectors):
material.SetMatrix("_CameraA_MATRIX_VP", camA.projectionMatrix * camA.worldToCameraMatrix);
As for getting the incident angle, it’s possible to do it with the projection matrix, but it requires transforming the vertex normal into projection space, which with the above setup requires transforming into world space first, so why do that extra work?
// vertex shader
o.worldNormal = UnityObjectToWorldNormal(v.normal);
o.worldPos = worldPos;
// fragment shader
float incidentA = dot(normalize(i.worldPos.xyz - _CameraAWorldPos.xyz), normalize(i.worldNormal));
float incidentB = dot(normalize(i.worldPos.xyz - _CameraBWorldPos.xyz), normalize(i.worldNormal));
