Phong Tessellation with separate displacement and lighting normals.

Unity Engine
1

Can someone give general advice on the best way to go about creating a shader that will do Phong tessellation using one set of normals, but allow for a separate normal channel for lighting? If possible, I would like to keep all the options available from the standard shader.

The use case is for something like an orange sliced in half. In this case, I’d like the mesh to be smoothed via tessellation, but if the triangles on the flat side of the slice have the correct normals for lighting (orthogonal to slice plane), then holes will appear due to the the tessellation at the seams going in different directions. If they share normals, then I lose out on lighting and geometric correctness. Any suggestions on the right approach for this?

2

Are you wanting dynamic slices, as in, it could be sliced in in any which way? If so, then as you slice, I would integrate this type of method.

Scalar tagged PN triangles. Otherwise, it’s basically a way of ‘tagging’ the vertexes to change how they are tessellated. Going by that line of thought, you could have a variable on each vertex specifying how it changes. You could change the smoothing to not smooth along those edges, or only smooth in certain directions.

You might be able to find an implementation of it directly, I’m not sure, but hopefully this leads you onto some thoughts for it!

3

Yes, I want to be able to dynamically slice (which is implemented already).

Thanks for the link! This does indeed seem relevant, though it is more abstract than what I am looking for. I actually know exactly what values I want the normals to be for both shading and tessellation at mesh generation time for each vertex. Unfortunately, I don’t have a lot of experience writing shaders so I’m not quite sure how to start a practical implementation of this, or how to hack it onto a standard shader. I’m not sure I even understand how I can even pass in additional normals to the shader. Unity seems to have additional optional UV channels (uv2, 3, 4) but no additional normal channel. Again, I don’t have a lot of experience writing shaders so maybe I am missing some simple common way to do things like this.

4

I’m guessing the way people normally do stuff like this is to use the vertex colors channel to store the data, which means destroying any vertex color functionality, which may be acceptable. In my case I do not need / want vertex colors, so this may work for me.

However, it seems now that I have two choices: I can try to modify phong tessellation to read the tessellation normals from the color channel, or I can modify everything else in the shader that needs a normal to read lighting normals from the color channel. Both of these seem challenging to me for injecting this functionality into something as complex as the standard shader.

For the first approach, I don’t really understand where I would go about modifying the phong tessellation shader logic. For instance, this reference:

uses the “tessphong” directive, but I have no idea if that is modifiable. I don’t see that term anywhere in the whole of the built in shader source code.

For the second approach, might there be some trick to easily modify the standard shader to get phong tessellation to displace the vertices based on the normal channel, but then assign the correct lighting normal before the rest of the shading logic runs?

Thanks in advance.

5

I would start with simpler shaders before trying to modify the standard shader. It’s rather complex. Either start with a surface shader (which automates lighting) or start with a simple vertex/fragment shader. Trying to start with the standard shader will get you frustrated very quickly.

Alternatively, an easy approach is to use the tool “Shader Forge” to get a shader put together. Especially, if you use the generated shaders, you can use them as a learning tool. It can quite easily reference all these parts of the mesh, and gives you a node-based structure to write the shader. Not free, but well worth the money if you’re interested in shaders.

The paper is quite abstract, but I also wanted to point out the idea of storing variables onto the mesh. There’s quite a lot of ways - you can even encode 4 ~8-bit floats into a single 32-bit float (with a loss of precision, so exact numbers are out, but approximates are fine generally) to turn the vertex colors into 16 variables. The same can be done with UV’s, and often times people do not use the second, third, and fourth UV channels.

But at the core, there’s two types of shaders - one is a surface shader, which is Unity-specific, and is compiled by a program called Shaderlab. It uses what you have and creates the underlying shaders for you. Alternatively, you can write a vertex/fragment, possibly with hull/domain/geometry sections, which is the core HLSL/GLSL shader, directly. The surface shader for phong tessellation you link there is based around the code-generation form, which is why you don’t see that mentioned. The Standard shader is a vertex/fragment shader.

On an additional note, the Standard shader does not include tessellation - and modifying it to support it would take quite a long time and a fair bit of shader experience (it’s not just modifying a few lines of code, but rather re-building the entire structure into a DX11 shader).

There’s a shader package called “Uber” that adds phong tessellation, though.

To be honest, you actually branched into a tricky subject. The easiest thing I could say to use is Shader Forge, I don’ think I have time to write up a shader to give an example. But if you want a learning experience, try writing simple vertex/fragment shaders first, get familiar with the mesh data and how it goes into a shader, and then see about adding tessellation to the shader.

6

Sounds like I have some knowledge gaps to fill before implementing this would be practical for me, but I think I follow mostly what is going on - thanks for your insight. Does Unity post the surface shader implementations anywhere so people can see what’s going on?

Thanks,
Ben

7

Not strictly, but,

And,

Also, almost all of the legacy shaders are surface shaders. Additionally, all the particle shaders are vertex/fragment shaders, so you can also look at those to get a basic idea of the alternative. Surface shaders are basically there to automate lighting for you.

I’ll send you a quick PM with some examples of my current shaders. They do a fair bit, you might be able to piece apart the most important bits though, and it might make any tessellation algorithms you find make more sense to see an example (they just have displacement, no phong). They aren’t surface shaders, but they’ll be useful to understand the pipeline, at least.

Oh, no PM’s allowed - send me one if you’d like to see the examples!

8

You should be able to send me a PM now. I’ve done some more experimenting and was able to successfully swap in normals from the color channels for lighting using a surface shader. However, unfortunately it seems like the “tessphong” pipeline (however that works) runs based on the mesh normals before the vertex function can modify them, so I don’t have the opportunity to change it unless there is some multiple pass trick I can take advantage of.

9

Sent you a PM. Those might make a little more sense. It’s not quite all there, so it won’t drop in and compile, but you’re not really missing anything of note (just remove the extra passes… back, back offset, the simple passes, etc… no need). Best of luck, though. Shoot me a reply if you need some more help.

10

Thanks!

11

I figured out a way to get the functionality I want for the most part, though I have no idea what the real performance implications are. Here is what I did:

  1. Create a surface shader that uses the Standard lighting model to get the PBR lighting (I don’t know the performance implications of this vs the real standard shader).
  2. Add the “tessphong” directive to get the Unity generated phong tessellation.
  3. Add “#pragma debug” to allow viewing the generated raw shader.
  4. Modify the raw shader’s code to replace the phong vertex displacement to use color channel instead of vertex. This requires changing the code in every (pass?) / multiple places. The relevant line to change in each tessellation snippet is:
for (int i = 0; i < 3; ++i)
    pp[i] = v.vertex.xyz - vi[i].normal* (dot(v.vertex.xyz, vi[i].normal) - dot(vi[i].vertex.xyz, vi[i].normal));

Get’s replaced with:

for (int i = 0; i < 3; ++i)
    pp[i] = v.vertex.xyz - vi[i].color * (dot(v.vertex.xyz, vi[i].color) - dot(vi[i].vertex.xyz, vi[i].color));

I was able to locate where to change without much shader knowledge simply by searching for the surface shader’s phong weighting variable (_Phong). Somewhat of an ugly work flow, but allows someone without a ton of shader knowledge to hack in a custom channel for phong tessellation to any surface shader.

Thanks for your help - you definitely got me on the path and in the right mindset to figure out what to do here. Here is surface shader I used as a base:

Shader "Custom/BenTestShader" {
        Properties {
            _EdgeLength ("Edge length", Range(2,50)) = 5
            _Phong ("Phong Strengh", Range(-1,1)) = 0
            _MainTex ("Base (RGB)", 2D) = "white" {}
            _NormalMap ("Normalmap", 2D) = "bump" {}
            _Color ("Color", color) = (1,1,1,0)
            _SpecPow ("Metallic", Range(0, 1)) = 0.5
            _GlossPow ("Smoothness", Range(0, 1)) = 0.5
            _EmissionColor("Color", Color) = (0,0,0)
            _EmissionMap("Emission", 2D) = "white" {}
        }
        SubShader {
            Tags { "RenderType"="Opaque" }
            LOD 300
           
            CGPROGRAM
            #pragma surface surf Standard addshadow fullforwardshadows vertex:disp tessellate:tessEdge tessphong:_Phong
            #pragma debug
            #include "Tessellation.cginc"

            struct appdata {
                float4 vertex : POSITION;
                float4 tangent : TANGENT;
                float3 normal : NORMAL;
                float3 color : COLOR;
                float2 texcoord : TEXCOORD0;
                float2 texcoord1 : TEXCOORD1;
                float2 texcoord2 : TEXCOORD2;
            };

            float _Phong;
            float _EdgeLength;
            half3 _EmissionColor;

            float4 tessEdge (appdata v0, appdata v1, appdata v2)
            {
                return UnityEdgeLengthBasedTess (v0.vertex, v1.vertex, v2.vertex, _EdgeLength);
            }

            sampler2D _DispTex;
            uniform float4 _DispTex_ST;
            sampler2D _EmissionMap;
            float _Displacement;
            float _DispOffset;
       
            void disp (inout appdata v)
            {
            }

            struct Input {
                float2 uv_MainTex;
                float2 uv_EmissionMap;
            };

            sampler2D _MainTex;
            sampler2D _NormalMap;
            fixed4 _Color;
            float _SpecPow;
            float _GlossPow;

            void surf (Input IN, inout SurfaceOutputStandard o) {
                half4 c = tex2D (_MainTex, IN.uv_MainTex) * _Color;
                o.Albedo = c.rgb;
                o.Metallic = _SpecPow;
                o.Smoothness = _GlossPow;
               
               // o.Normal = UnpackNormal(tex2D(_NormalMap, IN.uv_MainTex));
               
                o.Emission = tex2D (_MainTex, IN.uv_EmissionMap) * _EmissionColor;
            }
            ENDCG
        }
        FallBack "Diffuse"
    }
12

Glad you’ve got it working! That is a reasonable approach, yes. Editing the raw shader code is a bit messy, but I ‘think’ it compiles to a vertex / fragment shader there (or hull/domain, in this case)? I’ve always had issues opening compiled shaders with Unity. I either get a crash, I get the -really- compiled shader where it’s pseudo-assembly, or nothing happens at all.

If you’re ever interested in doing more with shaders, there is a ton you can do with careful manipulation of information between the meshes, textures, and more. It’s quite amazing how much the graphics and look of a game can be a pure result of shaders. Once you get that mindset, and especially once you get into the idea of easily modifying meshes through scripts (even buffers via compute shaders), a whole new world of flexibility opens up.