I have a bunch of 3d textMesh objects, each with a single letter on a single quad.
What I am trying to do with them is not exactly billboarding but the same idea: Putting the letter facing upward, then rotating it on y so its base is facing the camera, and tilting it up 10º on its local x axis to angle toward the camera a little more.
This works fine with a script that goes through every letter object every frame and says transform.LookAt (a world point below the main camera) but I know it is possible to do the same thing inside the shader and it would be much faster.
I’m comfortable enough with CG shaders now to do most of the color stuff I want to do, but I don’t understand all the model view projection matrix arcana and how they can be used to move verts around.
Yes, the modelView matrix is tricky to understand. Very, very roughly (I have done similar with non-Unity shaders, but it’s been a while):
Grab the 3DText shader (Font.shader.) We know that shows 3DText, and luckily, it has a vertex shader in it. Switch the material for your textMeshes to this. Change the vert shader to apply only the world position, but not the rotation. That will put the object in the right spot, but leave it in it’s “natural” rotation, which is probably facing the camera. Then look up “rotation matrix” (should be 3x3) and set up one for the 10 degree tilt. Arg, then apply the perspective matrix (all in the vert shader.) Look up how to apply a matrix to a vert, and the order, and how to chain them (it uses the MUL command, same as the original shader.)
I don’t think there’s a position-only matrix in Unity, but using the rightmost column of WorldView will give that (look up Translation matrix. It explains what the important numbers do and how to use them.) It may be the “natural” facing of a TextMesh is upwards, meaning you’ll see the edge, which is nothing. In that case, experiment with applying various rotation matrixes (maybe -90 on x?) to angle it forward. You may not be able to check in Scene View (the shader may keep twisting it away from you) so have to trial and error.
Of course, the optimization rule applies: if LookAt is fast enough, just leave it. Unless you really, really want a mini-project to learn shader matrixes.