VFACE (or SV_isFrontFace) can be used in the fragment shader (when surf is called), or written to by a geometry shader in DX10+. The value does not yet exist in the vertex shader as it gets calculated as part of the rasterization process which is between the vertex shader and the fragment shader, so trying to access it in “myvert” which is called during the vertex shader is going to just be either uninitialized memory or more likely always zero.
You’ll need to do a dot product against the view direction and surface normal, though unless you’re vertices are all unwelded and your vertex normals are aligned with the surface (ie: flat shaded) the results will be quite ugly. On something like a sphere you’ll get several vertices that are back facing and front facing within a single triangle causing it to get warped rather than having a clean “front” and “back” mesh. To do it properly would require using a geometry shader so you can have access to all 3 vertices of each triangle and computing a surface normal or enforcing the normal extrusion across all vertices of a triangle evenly. But Unity’s surface shaders do not support geometry shaders, so it would mean a fully custom vertex / fragment shader or hand modifying the generated code from a surface shader.
Basically I am just adding a #pragma multi_compile feature to my shader.
I was using VFACE so far, because I needed facing directiont on the fragment function level.
Now I need the facing direction it on the vertex function level - I guess I should define my custom variable based ondotProduct = dot(v.normal, viewDir) and then use it also in my fragment function to avoid the VFACE limitations.
Are there any advantages on using VFACE over dot product against the view direction and surface normal that I would be loosing?
A minor, but important correction, that’s a dot product of the vertex normal and the view direction. Even when done in the fragment shader it’s still an interpolation of the vertex normals, this is often referred to as the surface normal, but it isn’t actually. The true surface normal for each tri is a single constant direction across the entire tri, where as each of the vertex normals might be facing any direction.
Basically it’s possible (and quite likely on anything smoothed) that as a tri turns away from the camera some of the vertex normals of that one tri might have normals that are facing away from the camera and some that are facing towards. It’s even possible for all of the vertex normals to be facing away when the tri itself is still facing the camera, and vice versa. In short a dot product is actually a poor approximation of if a face is actually facing the camera or not as there’s little guarantee of the vertex normals matching the facing. So you might have all or just some of a tri being treated as flipped in the shader when it isn’t.
If you want to know if a tri is facing the camera, use VFACE.
If you want to know if a vertex normal is facing the camera, use a dot product. Also if you need a smooth transition, like a fresnel, you need to use the dot product too since VFACE is binary, even if it’s exact value differs between APIs (some do 0 or 1, some do false or true, some do -1 or 1, and some do 0x0000 or 0xFFFF)
Means it could be said that vertex normals include smoothing data, VFACE does not; i.e. if there is no smoothing the results should be the same, right?
One other advantages to using VFACE is bandwidth and shader performance. To use the dot product you need to pass the normal and view direction from the vertex shader to the fragment shader. That’s 6 floats ( actually 8 unless you do careful data packing) and there is a cost to to transferring that data. Additionally you have to do the dot product in the shader which is a few extra instructions. If your shader isn’t already needing / using that data elsewhere then it’ll be cheaper to use VFACE since it’s always being calculated during rasterization and is available to the fragment shader.
You can use VFACE in a vertex fragment shader as well. The compiled shader will use the appropriate semantic for the platform and translate it to a -1.0 or 1.0 value.
Really, all modern shader languages expect to be a bool like SV_IsFrontFacing. OpenGL’s equivalent gl_FrontFacing, Metal’s front_facing, and Vulkan’s FrontFacing are all bool values. Beyond that is really just the internal differences in how different shader languages handle bool values. You should be safe using SV_IsFrontFacing in your shaders regardless of the platform you compile to. Only the original Direct3D 9 VFACE was a -1 or 1 value, and Unity no longer supports D3D9.
Unfortunately Surface Shaders are stuck with VFACE since the feature was added when Unity still supported D3D9, so you’re stuck with an extra shader instruction if you don’t need that -1 or 1 float.
Is there a way to use VFACE in vertex shader?
And it seems the lightmap only can be fetch correctly for the front face. Is there a way to let back face pick up lightmap correctly, when back face is facing the light and front face is back to the light?
No. I thought my previous posts in this thread already covered that pretty well, but I’ll go into more detail as to why that doesn’t even make sense.
VFACE (or SV_IsFrontFacing) is whether or not the triangle being rendered is facing towards or away from the camera. The vertex shader has no knowledge of the triangle it might be used with at all, not to mention the facing of that triangle won’t be known until all vertex shaders for all 3 vertices of that triangle have finished running. And since you’re presumably wanting to use it in a still running vertex shader, that’s not exactly possible. Even ignoring that, each vertex may be used as the corner of any number of triangles in the mesh. It is possible (though highly unlikely outside of some fairly simple shapes) that one vertex is reused by every triangle in the mesh. So which “facing” would the vertex shader even use?
So, to reiterate for the third or fourth time in this thread:
You cannot useVFACEin the vertex shader. The closest you can get is by doing a dot product against the vertex normal with the view direction. But but that will never 100% match VFACE even in the best case of a flat shaded mesh were each vertex is only used with one triangle and the vertex normal matches the triangle facing.