DX11 Tessellation with vertex output, combination not working

[Acey195]

Hi,

Im currently working on creating more advanced shaders (in my opinion), but I am not a graphics programmer by trade.
The release of Unity 4, with DX11 was timed perfectly for my project, so I am very grateful for that and trying to use it to the greatest extent possible. But I am unable to combine my vertex/surface shader with the tessellation.

The tessellation works if I remove everything from my void surf() below “//mask” and my “, out Input o” from my void vert().
The problem is that I need this for my second procedural uv channel.
The shader also works without the “tessellate:tessEdge tessphong:_Smoothness” pragmas.

Anyone an idea what I am doing wrong? The warning I am getting is:

Shader warning in ‘Example/OceanShaderShort’: Program ‘tessvert_surf’, ‘vert’: no matching 1 parameter function (compiling for d3d11) at line 9

and some " implicit truncation of vector type" warnings which I don’t get when I turn off the tessellation.

Shader "Example/OceanShaderShort"
{
	Properties 
	{
		_Diffuse("_Diffuse", 2D) = "white" {}
		_StartBV ("Vert Blending start", float) = 5
		_EndBV ("Vert Blending end", float) = 1
		_StartBM ("Mask Blending start", float) = 6
		_EndBM ("Mask Blending end", float) = 1
		_StartBW ("Wave Blending start", float) = 7
		_EndBW ("Wave Blending end", float) = 1
		_TerrainScale("_TerrainScale", float) = 1

		_BaseTerrain("_UVmask", 2D) = "black" {}
		
		_TerrainTypeWater("_Terrain Type Water", 2D) = "black" {}
		_TerrainTypeWater_N("_Terrain Type Water_N", 2D) = "black" {}
		_TerrainTypeWater_S("Specular", Range(0,1)) = 0.1
		_TerrainTypeWater_SB("Specular Start", Range(0,1)) = 0.1
		
		_EdgeLength ("Edge length", Range(2,50)) = 5
        _Smoothness ("Phong Strengh", Range(0,1)) = 0.5
	}
	
	SubShader 
	{
		Tags
		{
			"Queue"="Transparent+1"
			"IgnoreProjector"="False"
			"RenderType"="Opaque"
		}
		
		Blend SrcAlpha OneMinusSrcAlpha
		Cull Back
		ZWrite Off
		ZTest LEqual
		ColorMask RGBA
		Fog{
	}

		CGPROGRAM
		#pragma exclude_renderers gles
		#pragma surface surf BlinnPhongEditor vertex:vert tessellate:tessEdge tessphong:_Smoothness
		#pragma target 5.0
		#include "Tessellation.cginc"
		#include "UnityCG.cginc"
	
			sampler2D _Diffuse;
			sampler2D _BaseTerrain;
			
			sampler2D _TerrainTypeWater;
			sampler2D _TerrainTypeWater_N;
									
			float _TerrainScale;
			
			float _TerrainTypeWater_S;
			float _TerrainTypeWater_SB;
			
			struct EditorSurfaceOutput {
				half3 Albedo;
				half3 Normal;
				half3 Emission;
				half3 Gloss;
				half Specular;
				half Alpha;
			};
			
			struct appdata {
				float4 vertex : POSITION;
            	float3 normal : NORMAL;
            	float2 texcoord : TEXCOORD0;
            	float2 texcoord1 : TEXCOORD1;
            	float4 tangent : TANGENT;
            };
			
			float _Smoothness;
			float _EdgeLength;

			float4 tessEdge (appdata v0, appdata v1, appdata v2)
			{
				return UnityEdgeLengthBasedTessCull (v0.vertex, v1.vertex, v2.vertex, _EdgeLength, 0.0);
			}
						
			struct Input {
				float4 meshUV;
				float3 blendVal;
			};

			float _StartBV;
			float _EndBV;
			float _StartBM;
			float _EndBM;
			float _StartBW;
			float _EndBW;
			
			void vert (inout appdata v, out Input o)
			{
				UNITY_INITIALIZE_OUTPUT(Input,o);
								
				float Pi = 3.1415926535;
				float4x4 rotationMatrix = float4x4(cos(0.5*Pi), -sin(0.5*Pi), 0.0, 0.0,
											sin(0.5*Pi), cos(0.5*Pi), 0.0, 0.0,
											0.0, 0.0, 1.0, 0.0,
											0.0, 0.0, 0.0, 1.0);

				
				float3 vertexR = mul(rotationMatrix,v.vertex.xyz) ;
				float radius = distance(vertexR.xyz, mul(_Object2World, float3(0,0,0)));
				float2 polarUV = float2(atan2(vertexR.y,vertexR.x),acos(vertexR.z/radius)*2);
				polarUV = polarUV*float2(0.1,0.1);
				
				o.meshUV.xy = (v.texcoord.xy);
				o.meshUV.zw = (polarUV);
				
				v.texcoord1.xy = polarUV;
				
		      	float distRangeV = clamp((_StartBV-(distance(mul (_Object2World, v.vertex).xyz, _WorldSpaceCameraPos.xyz)))*(1/(_StartBV-_EndBV)),0,1);
		      	float distRangeM = clamp((_StartBM-(distance(mul (_Object2World, v.vertex).xyz, _WorldSpaceCameraPos.xyz)))*(1/(_StartBM-_EndBM)),0,1);
		      	float distRangeW = clamp((_StartBW-(distance(mul (_Object2World, v.vertex).xyz, _WorldSpaceCameraPos.xyz)))*(1/(_StartBW-_EndBW)),0,1);
		      	
				o.blendVal.x = distRangeV;
				o.blendVal.y = distRangeM;
				o.blendVal.z = distRangeW;
			}
			
			inline half4 LightingBlinnPhongEditor_PrePass (EditorSurfaceOutput s, half4 light)
			{
				half3 spec = light.a * s.Gloss;
				half4 c;
				c.rgb = (s.Albedo * light.rgb + light.rgb * spec);
				c.a = s.Alpha;
								
				return c;
			}

			inline half4 LightingBlinnPhongEditor (EditorSurfaceOutput s, half3 lightDir, half3 viewDir, half atten)
			{
				half3 h = normalize (lightDir + viewDir);
				
				half diff = max (0, dot ( lightDir, s.Normal ));
				
				float nh = max (0, dot (s.Normal, h));
				float spec = pow (nh, s.Specular*128.0);
				
				half4 res;
				res.rgb = _LightColor0.rgb * diff;
				res.w = spec * Luminance (_LightColor0.rgb);
				res *= atten * 2.0;
				res *= s.Alpha;

				return LightingBlinnPhongEditor_PrePass( s, res );
			}
				
		
			void surf (Input IN, inout EditorSurfaceOutput o) 
			{
				o.Normal = float3(0.0,0.0,1.0);
				o.Alpha = 1;
				o.Albedo = 0.0;
				o.Emission = 0.0;
				o.Gloss = 0;
				o.Specular = 0;
				
				//mask
				float4 BlendP0 = tex2D(_BaseTerrain,(IN.meshUV).xy);
				
				//start diffuse
				float4 WaterA  = tex2D(_TerrainTypeWater,((IN.meshUV.xy)*_TerrainScale));
				float4 WaterB  = tex2D(_TerrainTypeWater,((IN.meshUV.zw)*_TerrainScale));
				float4 LerpPWat = lerp(WaterA,WaterB,BlendP0.w);

				//Normals
				float4 TerNwatA = tex2D(_TerrainTypeWater_N,((IN.meshUV.xy)*_TerrainScale));
				float3 UnpackNormalwatA = float3(UnpackNormal(TerNwatA));
				float4 TerNwatB = tex2D(_TerrainTypeWater_N,((IN.meshUV.zw)*_TerrainScale));
				float3 UnpackNormalwatB = float3(UnpackNormal(TerNwatB));
				float3 LerpPwatN = lerp(UnpackNormalwatA,UnpackNormalwatB,BlendP0.w);
										
				o.Albedo = LerpPWat.xyz;
				o.Normal = normalize(LerpPwatN);
				o.Alpha = IN.blendVal.x*lerp(BlendP0.y,BlendP0.x,IN.blendVal);
				o.Gloss = lerp(_TerrainTypeWater_SB,_TerrainTypeWater_S,IN.blendVal.x);
			}
		ENDCG
	}
	Fallback "Diffuse"
}

[RC-1290]

I remember running into the problem of not being able to combine a custom vertex shader function with the built in surface shader style tessellation, the last time I experimented with tessellation.

I worked around it by modifying the output of a surface shader with tessellation. But I ran out of time and didn’t do any further experiments.

Perhaps it makes sense to move the code that’s currently in the custom vertex shader function, to a custom domain shader function. (That way, it’ll be applied to all the vertices created by the tessellator, right?)

[Acey195]

Thanks for your time :), I just checked your water shader which is quite neat!

However I am a relative “newb” with shaders… could you expand on how you modified the output of your surface shader with tessellation?

Also you are correct that the vertex shader should also be applied on the new vertices, ’
but I have never built domain shaders, nor can I find much examples for them for unity. :neutral:

[RC-1290]

@Acey195

However I am a relative “newb” with shaders… could you expand on how you modified the output of your surface shader with tessellation?

I’m also a real amateur when it comes to tessellation. It’s been a while, so I can’t remember any of the details, but it basically came down to the following.
I compiled a really basic surface shader with tessellation, with #pragma debug to get an expanded version.

Shader "Custom/Testellate" {
    Properties {
        _MainTex ("Base (RGB)", 2D) = "white" {}
        _Offset ("Offset", Float) = 10
        _Tessellation ("Tessellation", Float) = 1
    }
    SubShader {
        Tags { "RenderType"="Opaque" }
        
        CGPROGRAM
        #pragma surface surf BlinnPhong vertex:displace tessellate:fixedTessellation nolightmap
        #pragma target 5.0
        #pragma debug
        
        sampler2D _MainTex;
        float _Offset;
        float _Tessellation;

        
        
        float fixedTessellation(){
            return _Tessellation;        
        }
        
        struct appdata {
            float4 vertex : POSITION;
//            float4 tangent : TANGENT;
            float3 normal : NORMAL;
            float4 texcoord : TEXCOORD0;
//            float4 texcoord1 : TEXCOORD1;
        };

        struct Input {
            float2 uv_MainTex;
        };
        
        void displace(inout appdata v){
//            UNITY_INITIALIZE_OUTPUT(Input,o);// Input Initialization required for DirectX11
        
            // Calculate mesh coordinates, 
            // Can't use TRANSFORM_TEX
            // since _HeightTex_ST does not seem to be available yet at this point in the shader
            // And setting  _HeightTex_ST gives problems when it is used later.
            // Does not actually change the texture coordinates.
            float4 customUVTransform = float4(1,1,0,0);
            float2 uv = v.texcoord.xy * customUVTransform.xy + customUVTransform.zw;
        
            float localTex = tex2Dlod(_MainTex, float4(uv,0,0)).r;
            
            v.vertex.y += localTex.r * _Offset;
//            v.normal = calcGridNormal(_HeightTex, uv, _HeightTex_TexelSize.xy, float3(1, _Offset, 1), localTex.r);// Store normal for lighting calculations
        }


        void surf (Input IN, inout SurfaceOutput o) {
            half4 c = tex2D (_MainTex, IN.uv_MainTex);
            o.Albedo = c.rgb;
        }
        ENDCG
    }
}

(There are neater examples in the Unity docs)
With the help of the MSDN docs information about Hull shaders and Domain Shaders, I pieced together a partially working set of ‘regular’ shaders (vertex, hull, domain and fragment)
The following is a shader that I found in my git repo. It seems to be the first compiling shader file with tessellation created in the way I described. It doesn’t seem to handle perspective correctly at all, but at least the basics work . Those problems are fixed in the next version I have but that version also contains all code for water rendering (I would like to say I could strip all that out for you, to get a version that purely shows the tessellation code, but I don’t think I have the time for that)
–[EDIT] This won’t render correctly. I recommend using the code I pasted a few posts down instead. –

Shader "Custom/BasicTessellation" {
    Properties {
        _MainTex ("Base (RGB)", 2D) = "white" {}
        _Offset ("Offset", Float) = 10
        _Tess ("Tessellation", Float) = 2
    }
    SubShader {
        Tags { "RenderType"="Opaque" }
        Pass {
        
            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag
            #pragma hull tessBase
            #pragma domain basicDomain
            #pragma target 5.0
            #include "UnityCG.cginc"
            #define INTERNAL_DATA
    
            sampler2D _MainTex;
            
            float4 _MainTex_ST;
            float _Offset;
            float _Tess;
            
            struct v2f{
                float4 pos : SV_POSITION;
                float2 texcoord : TEXCOORD0;
            };
            
            #ifdef UNITY_CAN_COMPILE_TESSELLATION
            
            struct inputControlPoint{
                float4 position : WORLDPOS;
                float4 texcoord : TEXCOORD0;
                float4 tangent : TANGENT;
                float3 normal : NORMAL;
            };
            
            struct outputControlPoint{
                float3 position : BEZIERPOS;            
            };
            
            struct outputPatchConstant{
                float edges[3]        : SV_TessFactor;
                float inside        : SV_InsideTessFactor;
                
                float3 vTangent[4]    : TANGENT;
                float2 vUV[4]         : TEXCOORD;
                float3 vTanUCorner[4] : TANUCORNER;
                float3 vTanVCorner[4] : TANVCORNER;
                float4 vCWts          : TANWEIGHTS;
            };
            
            
            outputPatchConstant patchConstantThing(InputPatch<inputControlPoint, 3> v){
                outputPatchConstant o;
                
                o.edges[0] = _Tess;
                o.edges[1] = _Tess;
                o.edges[2] = _Tess;
                o.inside = _Tess;
                
                return o;
            }
            
            // tessellation hull shader
            [domain("tri")]
            [partitioning("fractional_odd")]
            [outputtopology("triangle_cw")]
            [patchconstantfunc("patchConstantThing")]
            [outputcontrolpoints(3)]
            inputControlPoint tessBase (InputPatch<inputControlPoint,3> v, uint id : SV_OutputControlPointID) {
                return v[id];
            }
            
            #endif // UNITY_CAN_COMPILE_TESSELLATION
            
            v2f vert (appdata_tan v){
                v2f o;        
                
                o.texcoord = TRANSFORM_TEX (v.texcoord, _MainTex);
                
                float localTex = tex2Dlod(_MainTex, float4(o.texcoord,0,0)).r;
                v.vertex.y += localTex.r * _Offset;
                
                o.pos = mul(UNITY_MATRIX_MVP,v.vertex);
                
                return o;
            }
            
            #ifdef UNITY_CAN_COMPILE_TESSELLATION
            
            // tessellation domain shader
            [domain("tri")]
            v2f basicDomain (outputPatchConstant tessFactors, const OutputPatch<inputControlPoint,3> vi, float3 bary : SV_DomainLocation) {
                appdata_tan v;
                v.vertex = vi[0].position*bary.x + vi[1].position*bary.y + vi[2].position*bary.z;
                v.tangent = vi[0].tangent*bary.x + vi[1].tangent*bary.y + vi[2].tangent*bary.z;
                v.normal = vi[0].normal*bary.x + vi[1].normal*bary.y + vi[2].normal*bary.z;
                v.texcoord = vi[0].texcoord*bary.x + vi[1].texcoord*bary.y + vi[2].texcoord*bary.z;
                v2f o = vert( v);
//                v2f o = vert_surf (v);
                return o;
            }
            
            #endif // UNITY_CAN_COMPILE_TESSELLATION
            
            
    
            float4 frag(in v2f IN):COLOR{
                return tex2D (_MainTex, IN.texcoord);
            }
            
            ENDCG
        }
    } 
}

It might not be a useful shader on its own ^^, but this shader does show a basic working framework with the almost all shader stages present (there’s no geometry shader in this one).

@Acey195

Also you are correct that the vertex shader should also be applied on the new vertices, ’
but I have never built domain shaders, nor can I find much examples for them for unity.

Yeah, back in January I couldn’t find much helpful information either. But if you are better at reading than me, the documentation on MSDN should be a real help. And even if you’re not, those articles contain enough information to allow you to piece it together. Most of the time you can ignore all the CPU code (setting rendering state, stuff like that), since Unity handles that stuff for us. So that means, ignore the ‘how to create(…)’ articles, and focus on the ‘how to design (…)’ articles.

Does that help?

[Acey195]

Thank you very much! :), I will look through these and if I can still not find a solution, I think I have give up my “fancy wave system” idea :P…

I would be very interested in that water shader though (or a stripped version), maybe we could trade it for something XD? I do procedural art ;),
as well as modeling.

I am afraid the “Custom/BasicTessellation” shader does not really work, it does compile, but makes the object invisible and scales it to a vertical line.
I will try to get it working though, but so far I haven’t got it yet.

[RC-1290]

Hmmm, It’s probably because in that version I still used the vertex shader function in the domain shader. So the position is multiplied with the MVP matrix too many times.

Try this:

Shader "Custom/BasicTessellation" {
    Properties {
        _MainTex ("Base (RGB)", 2D) = "white" {}
        _Offset ("Offset", Float) = 10
        _Tess ("Tessellation", Float) = 2
    }
    SubShader {
        Tags { "RenderType"="Opaque" }
        Pass {
        
            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag
            #pragma hull tessBase
            #pragma domain basicDomain
            #pragma target 5.0
            #include "UnityCG.cginc"
            #define INTERNAL_DATA
    
            sampler2D _MainTex;
            
            float4 _MainTex_ST;
            float _Offset;
            float _Tess;
            
            struct v2f{
                float4 pos : SV_POSITION;
                float2 texcoord : TEXCOORD0;
            };
            
            #ifdef UNITY_CAN_COMPILE_TESSELLATION
            
            struct inputControlPoint{
                float4 position : WORLDPOS;
                float4 texcoord : TEXCOORD0;
                float4 tangent : TANGENT;
                float3 normal : NORMAL;
            };
            
            struct outputControlPoint{
                float3 position : BEZIERPOS;            
            };
            
            struct outputPatchConstant{
                float edges[3]        : SV_TessFactor;
                float inside        : SV_InsideTessFactor;
                
                float3 vTangent[4]    : TANGENT;
                float2 vUV[4]         : TEXCOORD;
                float3 vTanUCorner[4] : TANUCORNER;
                float3 vTanVCorner[4] : TANVCORNER;
                float4 vCWts          : TANWEIGHTS;
            };
            
            
            outputPatchConstant patchConstantThing(InputPatch<inputControlPoint, 3> v){
                outputPatchConstant o;
                
                o.edges[0] = _Tess;
                o.edges[1] = _Tess;
                o.edges[2] = _Tess;
                o.inside = _Tess;
                
                return o;
            }
            
            // tessellation hull shader
            [domain("tri")]
            [partitioning("fractional_odd")]
            [outputtopology("triangle_cw")]
            [patchconstantfunc("patchConstantThing")]
            [outputcontrolpoints(3)]
            inputControlPoint tessBase (InputPatch<inputControlPoint,3> v, uint id : SV_OutputControlPointID) {
                return v[id];
            }
            
            #endif // UNITY_CAN_COMPILE_TESSELLATION
            
            v2f vert (appdata_tan v){
                v2f o;
                
                o.texcoord = v.texcoord;
                o.pos = v.vertex;
                
                return o;
            }
            
            v2f displace (appdata_tan v){
                v2f o;        
                
                o.texcoord = TRANSFORM_TEX (v.texcoord, _MainTex);
                
                float localTex = tex2Dlod(_MainTex, float4(o.texcoord,0,0)).r;
                v.vertex.y += localTex.r * _Offset;
                
                o.pos = mul(UNITY_MATRIX_MVP,v.vertex);
                
                return o;
            }
            
            #ifdef UNITY_CAN_COMPILE_TESSELLATION
            
            // tessellation domain shader
            [domain("tri")]
            v2f basicDomain (outputPatchConstant tessFactors, const OutputPatch<inputControlPoint,3> vi, float3 bary : SV_DomainLocation) {
                appdata_tan v;
                v.vertex = vi[0].position*bary.x + vi[1].position*bary.y + vi[2].position*bary.z;
                v.tangent = vi[0].tangent*bary.x + vi[1].tangent*bary.y + vi[2].tangent*bary.z;
                v.normal = vi[0].normal*bary.x + vi[1].normal*bary.y + vi[2].normal*bary.z;
                v.texcoord = vi[0].texcoord*bary.x + vi[1].texcoord*bary.y + vi[2].texcoord*bary.z;
                v2f o = displace( v);
//                v2f o = vert_surf (v);
                return o;
            }
            
            #endif // UNITY_CAN_COMPILE_TESSELLATION
            
            
    
            float4 frag(in v2f IN):COLOR{
                return tex2D (_MainTex, IN.texcoord);
            }
            
            ENDCG
        }
    } 
}

[RC-1290]

@Acey195

I would be very interested in that water shader though (or a stripped version), maybe we could trade it for something XD? I do procedural art ;),
as well as modeling.

I’m currently working on making the whole water simulation usable as tool, and I want to make it available on the asset store. So that would also include the water shaders. But I’m afraid it wouldn’t exactly be cheap, considering a student’s budget (I saw you’re attending NHTV here in The Netherlands ). So, hopefully the latest version of the shader I just posted is more useful.

[Acey195]

Im afraid the new shader gives the same problem :(,
it squeezes the model to a point/line depending on the offset.

I already saw you were Dutch, pretty cool I guess :p.
But I understand how you feel about your project.
I am working on a project of my own as well, which I feel similarly about

[RC-1290]

Are you sure Unity is set to DX11 mode? Do the vertex and fragment shaders show up as shader model 5?
Resets the shader to test default settings

Ahhh, did I mention that it requires a heightmap? XD
I’m using a terrain heightmap, but for testing purposes you should be able to use any texture. It only cares about the red channel.

[Acey195]

I am pretty sure I run in DX11, as I already use Shader model 5 shaders for other stuff and I also did apply several textures, but they all seem to give the same result for me somehow

I’m currently using Unity 4.0.1.f2

[RC-1290]

Interesting…
And the surface shader version does work?
This is what it looks like for me:

Does it work if you remove the displacement code?

float localTex = tex2Dlod(_MainTex, float4(o.texcoord,0,0)).r;
v.vertex.y += localTex.r * _Offset;

[Acey195]

hmm well over here it looks like this (I renamed the shader):

Also the sampling from the texture works fine (I guess) as with different textures the length of the line and starting height differ. Turning that off, is the same as setting the offset to 0.

I also really appreciate the help, I think I can send you my first asset store pack (which I just sent for approval) with object space normal map shaders, if you want those of course ;).

Maybe you could send me a unity package with the scene shown in the image above? I really don’t see what I am doing wrong here

Edit: I just upgraded to Unity 4.1, but that also did not change anything. (even though there were shader updates in this version)

Edit, I found a different Fragment based Tessellation that does work for me :

Shader "Tessellation/Simple Tess PN Disp" {
Properties {
	_MainTex ("Main Texture", 2D) = "white" {}
	_DispTex ("Disp Texture", 2D) = "gray" {}
	_TessEdge ("Edge Tess", Range(1,16)) = 2
	_Displacement ("Displacement", Range(0, 0.3)) = 0.1
}
SubShader {
Pass {
	Tags {"LightMode" = "Vertex"}

CGPROGRAM
#pragma target 5.0

//#pragma vertex VS_RenderScene
//#pragma fragment PS_RenderSceneTextured

#pragma vertex VS_RenderSceneWithTessellation
#pragma fragment PS_RenderSceneTextured
#pragma hull HS_PNTriangles
#pragma domain DS_PNTriangles

#include "UnityCG.cginc"

float _TessEdge;
float _Displacement;


//=================================================================================================================================
// Buffers, Textures and Samplers
//=================================================================================================================================

Texture2D _MainTex;
SamplerState	sampler_MainTex;
Texture2D _DispTex;
SamplerState	sampler_DispTex;

//=================================================================================================================================
// Shader structures
//=================================================================================================================================


struct VS_RenderSceneInput
{
    float3 vertex : POSITION;
    float3 normal : NORMAL;
	float2 texcoord : TEXCOORD0;
};

struct HS_Input
{
    float4 f4Position   : POS;
    float3 f3Normal     : NORMAL;
    float2 f2TexCoord   : TEXCOORD;
};

struct HS_ConstantOutput
{
    // Tess factor for the FF HW block
    float fTessFactor[3]    : SV_TessFactor;
    float fInsideTessFactor : SV_InsideTessFactor;
    
    // Geometry cubic generated control points
    float3 f3B210    : POS3;
    float3 f3B120    : POS4;
    float3 f3B021    : POS5;
    float3 f3B012    : POS6;
    float3 f3B102    : POS7;
    float3 f3B201    : POS8;
    float3 f3B111    : CENTER;
    
    // Normal quadratic generated control points
    float3 f3N110    : NORMAL3;      
    float3 f3N011    : NORMAL4;
    float3 f3N101    : NORMAL5;
};

struct HS_ControlPointOutput
{
    float3    f3Position    : POS;
    float3    f3Normal      : NORMAL;
    float2    f2TexCoord    : TEXCOORD;
};

struct DS_Output
{
    float4 f4Position   : SV_Position;
    float2 f2TexCoord   : TEXCOORD0; 
    float4 f4Diffuse    : COLOR0;
};

struct PS_RenderSceneInput
{
    float4 f4Position   : SV_Position;
    float2 f2TexCoord   : TEXCOORD0;
    float4 f4Diffuse    : COLOR0;
};

struct PS_RenderOutput
{
    float4 f4Color      : SV_Target0;
};

PS_RenderSceneInput VS_RenderScene( VS_RenderSceneInput I )
{
    PS_RenderSceneInput O;
    
    O.f4Position = mul (UNITY_MATRIX_MVP, float4(I.vertex, 1.0f));
	float3 viewN = mul ((float3x3)UNITY_MATRIX_IT_MV, I.normal);
    
    // Calc diffuse color    
    O.f4Diffuse.rgb = unity_LightColor[0].rgb * max(0, dot(viewN, unity_LightPosition[0].xyz)) + UNITY_LIGHTMODEL_AMBIENT.rgb;
    O.f4Diffuse.a = 1.0f;
    
    // Pass through texture coords
    O.f2TexCoord = I.texcoord; 
    
    return O;    
}


HS_Input VS_RenderSceneWithTessellation( VS_RenderSceneInput I )
{
    HS_Input O;
	
	// To view space
    O.f4Position = mul(UNITY_MATRIX_MV, float4(I.vertex,1.0f));
	O.f3Normal = mul ((float3x3)UNITY_MATRIX_IT_MV, I.normal);
        
    O.f2TexCoord = I.texcoord;
    
    return O;
}


//=================================================================================================================================
// This hull shader passes the tessellation factors through to the HW tessellator, 
// and the 10 (geometry), 6 (normal) control points of the PN-triangular patch to the domain shader
//=================================================================================================================================
HS_ConstantOutput HS_PNTrianglesConstant( InputPatch<HS_Input, 3> I )
{
    HS_ConstantOutput O = (HS_ConstantOutput)0;
    
    // Simply output the tessellation factors from constant space 
    // for use by the FF tessellation unit
	O.fTessFactor[0] = O.fTessFactor[1] = O.fTessFactor[2] = _TessEdge;
	O.fInsideTessFactor = _TessEdge;

    // Assign Positions
    float3 f3B003 = I[0].f4Position.xyz;
    float3 f3B030 = I[1].f4Position.xyz;
    float3 f3B300 = I[2].f4Position.xyz;
    // And Normals
    float3 f3N002 = I[0].f3Normal;
    float3 f3N020 = I[1].f3Normal;
    float3 f3N200 = I[2].f3Normal;
        
    // Compute the cubic geometry control points
    // Edge control points
    O.f3B210 = ( ( 2.0f * f3B003 ) + f3B030 - ( dot( ( f3B030 - f3B003 ), f3N002 ) * f3N002 ) ) / 3.0f;
    O.f3B120 = ( ( 2.0f * f3B030 ) + f3B003 - ( dot( ( f3B003 - f3B030 ), f3N020 ) * f3N020 ) ) / 3.0f;
    O.f3B021 = ( ( 2.0f * f3B030 ) + f3B300 - ( dot( ( f3B300 - f3B030 ), f3N020 ) * f3N020 ) ) / 3.0f;
    O.f3B012 = ( ( 2.0f * f3B300 ) + f3B030 - ( dot( ( f3B030 - f3B300 ), f3N200 ) * f3N200 ) ) / 3.0f;
    O.f3B102 = ( ( 2.0f * f3B300 ) + f3B003 - ( dot( ( f3B003 - f3B300 ), f3N200 ) * f3N200 ) ) / 3.0f;
    O.f3B201 = ( ( 2.0f * f3B003 ) + f3B300 - ( dot( ( f3B300 - f3B003 ), f3N002 ) * f3N002 ) ) / 3.0f;
    // Center control point
    float3 f3E = ( O.f3B210 + O.f3B120 + O.f3B021 + O.f3B012 + O.f3B102 + O.f3B201 ) / 6.0f;
    float3 f3V = ( f3B003 + f3B030 + f3B300 ) / 3.0f;
    O.f3B111 = f3E + ( ( f3E - f3V ) / 2.0f );
    
    // Compute the quadratic normal control points
    float fV12 = 2.0f * dot( f3B030 - f3B003, f3N002 + f3N020 ) / dot( f3B030 - f3B003, f3B030 - f3B003 );
    O.f3N110 = normalize( f3N002 + f3N020 - fV12 * ( f3B030 - f3B003 ) );
    float fV23 = 2.0f * dot( f3B300 - f3B030, f3N020 + f3N200 ) / dot( f3B300 - f3B030, f3B300 - f3B030 );
    O.f3N011 = normalize( f3N020 + f3N200 - fV23 * ( f3B300 - f3B030 ) );
    float fV31 = 2.0f * dot( f3B003 - f3B300, f3N200 + f3N002 ) / dot( f3B003 - f3B300, f3B003 - f3B300 );
    O.f3N101 = normalize( f3N200 + f3N002 - fV31 * ( f3B003 - f3B300 ) );
           
    return O;
}

[domain("tri")]
[partitioning("fractional_odd")]
[outputtopology("triangle_cw")]
[patchconstantfunc("HS_PNTrianglesConstant")]
[outputcontrolpoints(3)]
HS_ControlPointOutput HS_PNTriangles( InputPatch<HS_Input, 3> I, uint uCPID : SV_OutputControlPointID )
{
    HS_ControlPointOutput O = (HS_ControlPointOutput)0;

    // Just pass through inputs = fast pass through mode triggered
    O.f3Position = I[uCPID].f4Position.xyz;
    O.f3Normal = I[uCPID].f3Normal;
    O.f2TexCoord = I[uCPID].f2TexCoord;
    
    return O;
}


//=================================================================================================================================
// This domain shader applies contol point weighting to the barycentric coords produced by the FF tessellator 
//=================================================================================================================================
[domain("tri")]
DS_Output DS_PNTriangles( HS_ConstantOutput HSConstantData, const OutputPatch<HS_ControlPointOutput, 3> I, float3 f3BarycentricCoords : SV_DomainLocation )
{
    DS_Output O = (DS_Output)0;

    // The barycentric coordinates
    float fU = f3BarycentricCoords.x;
    float fV = f3BarycentricCoords.y;
    float fW = f3BarycentricCoords.z;

    // Precompute squares and squares * 3 
    float fUU = fU * fU;
    float fVV = fV * fV;
    float fWW = fW * fW;
    float fUU3 = fUU * 3.0f;
    float fVV3 = fVV * 3.0f;
    float fWW3 = fWW * 3.0f;
    
    // Compute position from cubic control points and barycentric coords
    float3 f3Position = I[0].f3Position * fWW * fW +
                        I[1].f3Position * fUU * fU +
                        I[2].f3Position * fVV * fV +
                        HSConstantData.f3B210 * fWW3 * fU +
                        HSConstantData.f3B120 * fW * fUU3 +
                        HSConstantData.f3B201 * fWW3 * fV +
                        HSConstantData.f3B021 * fUU3 * fV +
                        HSConstantData.f3B102 * fW * fVV3 +
                        HSConstantData.f3B012 * fU * fVV3 +
                        HSConstantData.f3B111 * 6.0f * fW * fU * fV;
    
    // Compute normal from quadratic control points and barycentric coords
    float3 f3Normal =   I[0].f3Normal * fWW +
                        I[1].f3Normal * fUU +
                        I[2].f3Normal * fVV +
                        HSConstantData.f3N110 * fW * fU +
                        HSConstantData.f3N011 * fU * fV +
                        HSConstantData.f3N101 * fW * fV;

    // Normalize the interpolated normal    
    f3Normal = normalize( f3Normal );

    // Linearly interpolate the texture coords
    O.f2TexCoord = I[0].f2TexCoord * fW + I[1].f2TexCoord * fU + I[2].f2TexCoord * fV;
	
	// Displacement map!
	float disp = _DispTex.SampleLevel (sampler_DispTex, O.f2TexCoord, 0).r * _Displacement;
	f3Position += f3Normal * disp;

    // Calc diffuse color    
    O.f4Diffuse.rgb = unity_LightColor[0].rgb * max( 0, dot( f3Normal, unity_LightPosition[0].xyz ) ) + UNITY_LIGHTMODEL_AMBIENT.rgb;
    O.f4Diffuse.a = 1.0f;

    // Transform position with projection matrix
    O.f4Position = mul (UNITY_MATRIX_P, float4(f3Position.xyz,1.0));
        
    return O;
}


PS_RenderOutput PS_RenderSceneTextured( PS_RenderSceneInput I )
{
    PS_RenderOutput O;
    
    O.f4Color = _MainTex.Sample( sampler_MainTex, I.f2TexCoord ).r * I.f4Diffuse;
    
    return O;
}

ENDCG

}
}

Fallback "VertexLit"
}

I also realized that this shader (as well as yours) are not camera distance based, which would be preferable. I will try to look into that as well.

[RC-1290]

Now I’m really curious about what’s going on… I don’t remember seeing you need Unity Pro for Tessellation, or anything like that.
If I ever want to use DX11 tessellation for dynamic level of detail in terrain and water meshes, I need to know why it doesn’t work on your machine. So I’ve created a test scene to figure out where it’s exploding.

This is what it looks like on my machine (OH wow, the attached image actually worked immediately this time )

Left: Example script from documentation.
Middle: Mutilated example script I used to figure out tessellation
Right: Reconstructed Vertex, Hull, Domain, Fragment shader version

Here’s the scene: 1192474–46973–$TessellationExperiments.unitypackage (266 KB)

If you could upload a screenshot of what it looks like for you, that might help me figure out what’s going wrong. The package includes a basic debug information script that hopefully gives some insight into the differences between your setup and mine. (I didn’t create it specifically for this scene though; we probably don’t need to know if your pc supports ARGBFloat RenderTextures :p)

[Edit] I saw your edit too late ^^. Well, good news

[Acey195]

Thank you very much I just opened you scene,
the interesting part: 2 out of 3 work:

These work:
Tessellation Docs Sample
Surface Shader Tessellation

Still give the same problem:
Vertex Hull Domain Fragment Tessellation

My setup does not support: RenderTextures, ARGBFloat and Depth Textures (pro features I guess)
The rest is all supported. There is a possibility that it is because I use an Nvidia card, where you have an ATI card,
but your water demo-build did work on my machine. Or maybe a missing DLL, I really don’t know XD.

I also threw together a distance based tesselation thing:

This was done by editing the following function in the shader in my last post.
Note that the current implementation creates holes in the geometry
and the conversion matrix for the distance is incorrect.
(If you scale or move the object, it won’t work anymore)
and the blend distances are hardcoded for now.

Do you think it is even possible to get this seamless,
or will this require the use of a surface shader by default?

HS_ConstantOutput HS_PNTrianglesConstant( InputPatch<HS_Input, 3> I )
{
    HS_ConstantOutput O = (HS_ConstantOutput)0;
    
    // Simply output the tessellation factors from constant space 
    // for use by the FF tessellation unit

    
    // Assign Positions
    float3 f3B003 = I[0].f4Position.xyz;
    float3 f3B030 = I[1].f4Position.xyz;
    float3 f3B300 = I[2].f4Position.xyz;
    // And Normals
    float3 f3N002 = I[0].f3Normal;
    float3 f3N020 = I[1].f3Normal;
    float3 f3N200 = I[2].f3Normal;
    
    float _StartBV = 3;
    float _EndBV = 0;
    float3 midPos = mul(_Object2World,(f3B003+f3B030+f3B300)/3);
	float3 camPos = mul(_World2Object,_WorldSpaceCameraPos.xyz); 
    //float camDist = clamp((_StartBV-(distance(mul (_Object2World, midPos).xyz, _WorldSpaceCameraPos.xyz)))*(1/(_StartBV-_EndBV)),0,1);
    //float camDist = clamp((_StartBV-(distance(mul(_Object2World, midPos).xyz, _WorldSpaceCameraPos.xyz)))*(1/(_StartBV-_EndBV)),0,1);
    
    
    float camDist = clamp(_StartBV-(distance(midPos.xyz, camPos.xyz))*(1/(_StartBV-_EndBV)),0,1);
    //float camDist = distance(mul(_Object2World, midPos).xyz, _WorldSpaceCameraPos.xyz);
    
	O.fTessFactor[0] = O.fTessFactor[1] = O.fTessFactor[2] = max(camDist*_TessEdge,1);
	O.fInsideTessFactor = max(camDist*_TessEdge,1);
        
    // Compute the cubic geometry control points
    // Edge control points
    O.f3B210 = ( ( 2.0f * f3B003 ) + f3B030 - ( dot( ( f3B030 - f3B003 ), f3N002 ) * f3N002 ) ) / 3.0f;
    O.f3B120 = ( ( 2.0f * f3B030 ) + f3B003 - ( dot( ( f3B003 - f3B030 ), f3N020 ) * f3N020 ) ) / 3.0f;
    O.f3B021 = ( ( 2.0f * f3B030 ) + f3B300 - ( dot( ( f3B300 - f3B030 ), f3N020 ) * f3N020 ) ) / 3.0f;
    O.f3B012 = ( ( 2.0f * f3B300 ) + f3B030 - ( dot( ( f3B030 - f3B300 ), f3N200 ) * f3N200 ) ) / 3.0f;
    O.f3B102 = ( ( 2.0f * f3B300 ) + f3B003 - ( dot( ( f3B003 - f3B300 ), f3N200 ) * f3N200 ) ) / 3.0f;
    O.f3B201 = ( ( 2.0f * f3B003 ) + f3B300 - ( dot( ( f3B300 - f3B003 ), f3N002 ) * f3N002 ) ) / 3.0f;
    // Center control point
    float3 f3E = ( O.f3B210 + O.f3B120 + O.f3B021 + O.f3B012 + O.f3B102 + O.f3B201 ) / 6.0f;
    float3 f3V = ( f3B003 + f3B030 + f3B300 ) / 3.0f;
    O.f3B111 = f3E + ( ( f3E - f3V ) / 2.0f );
    
    // Compute the quadratic normal control points
    float fV12 = 2.0f * dot( f3B030 - f3B003, f3N002 + f3N020 ) / dot( f3B030 - f3B003, f3B030 - f3B003 );
    O.f3N110 = normalize( f3N002 + f3N020 - fV12 * ( f3B030 - f3B003 ) );
    float fV23 = 2.0f * dot( f3B300 - f3B030, f3N020 + f3N200 ) / dot( f3B300 - f3B030, f3B300 - f3B030 );
    O.f3N011 = normalize( f3N020 + f3N200 - fV23 * ( f3B300 - f3B030 ) );
    float fV31 = 2.0f * dot( f3B003 - f3B300, f3N200 + f3N002 ) / dot( f3B003 - f3B300, f3B003 - f3B300 );
    O.f3N101 = normalize( f3N200 + f3N002 - fV31 * ( f3B003 - f3B300 ) );
           
    return O;
}

Also my offer still counts, if you want my object space normal shaders (which don’t plan to be free for everyone :P)

[RC-1290]

My water simulation doesn’t use tessellation yet, (which is also why it’ll run slow if you run it at resolutions like 2048 * 2048, on a graphics card like mine).
I think you might be right when you say it could be because of the difference between AMD and Nvidia cards; I probably forgot to include something that’s required on Nvidia cards.

The surface shader version is first compiled to regular shaders first, so anything you could do with a surface shader should be possible to do manually. But it requires more work and knowledge of the underlying shader stages. Unfortunately, I don’t yet know how to make sure both versions of edges are subdivided by the same amount, so I can’t tell you how to fix those gaps.

Have you taken a look at how the distance-based tessellation surface shader example from the docs compiles?

Thanks for the offer, by the way! I’ve never needed object space normal mapped objects before, but perhaps I’d learn something from those shaders.

[Marionette]

@Acey195

Thank you very much I just opened you scene,
the interesting part: 2 out of 3 work:

These work:
Tessellation Docs Sample
Surface Shader Tessellation

Still give the same problem:
Vertex Hull Domain Fragment Tessellation

My setup does not support: RenderTextures, ARGBFloat and Depth Textures (pro features I guess)
The rest is all supported. There is a possibility that it is because I use an Nvidia card, where you have an ATI card,
but your water demo-build did work on my machine. Or maybe a missing DLL, I really don’t know XD.

I also threw together a distance based tesselation thing:

This was done by editing the following function in the shader in my last post.
Note that the current implementation creates holes in the geometry
and the conversion matrix for the distance is incorrect.
(If you scale or move the object, it won’t work anymore)
and the blend distances are hardcoded for now.

Do you think it is even possible to get this seamless,
or will this require the use of a surface shader by default?

HS_ConstantOutput HS_PNTrianglesConstant( InputPatch<HS_Input, 3> I )
{
    HS_ConstantOutput O = (HS_ConstantOutput)0;
    
    // Simply output the tessellation factors from constant space 
    // for use by the FF tessellation unit

    
    // Assign Positions
    float3 f3B003 = I[0].f4Position.xyz;
    float3 f3B030 = I[1].f4Position.xyz;
    float3 f3B300 = I[2].f4Position.xyz;
    // And Normals
    float3 f3N002 = I[0].f3Normal;
    float3 f3N020 = I[1].f3Normal;
    float3 f3N200 = I[2].f3Normal;
    
    float _StartBV = 3;
    float _EndBV = 0;
    float3 midPos = mul(_Object2World,(f3B003+f3B030+f3B300)/3);
	float3 camPos = mul(_World2Object,_WorldSpaceCameraPos.xyz); 
    //float camDist = clamp((_StartBV-(distance(mul (_Object2World, midPos).xyz, _WorldSpaceCameraPos.xyz)))*(1/(_StartBV-_EndBV)),0,1);
    //float camDist = clamp((_StartBV-(distance(mul(_Object2World, midPos).xyz, _WorldSpaceCameraPos.xyz)))*(1/(_StartBV-_EndBV)),0,1);
    
    
    float camDist = clamp(_StartBV-(distance(midPos.xyz, camPos.xyz))*(1/(_StartBV-_EndBV)),0,1);
    //float camDist = distance(mul(_Object2World, midPos).xyz, _WorldSpaceCameraPos.xyz);
    
	O.fTessFactor[0] = O.fTessFactor[1] = O.fTessFactor[2] = max(camDist*_TessEdge,1);
	O.fInsideTessFactor = max(camDist*_TessEdge,1);
        
    // Compute the cubic geometry control points
    // Edge control points
    O.f3B210 = ( ( 2.0f * f3B003 ) + f3B030 - ( dot( ( f3B030 - f3B003 ), f3N002 ) * f3N002 ) ) / 3.0f;
    O.f3B120 = ( ( 2.0f * f3B030 ) + f3B003 - ( dot( ( f3B003 - f3B030 ), f3N020 ) * f3N020 ) ) / 3.0f;
    O.f3B021 = ( ( 2.0f * f3B030 ) + f3B300 - ( dot( ( f3B300 - f3B030 ), f3N020 ) * f3N020 ) ) / 3.0f;
    O.f3B012 = ( ( 2.0f * f3B300 ) + f3B030 - ( dot( ( f3B030 - f3B300 ), f3N200 ) * f3N200 ) ) / 3.0f;
    O.f3B102 = ( ( 2.0f * f3B300 ) + f3B003 - ( dot( ( f3B003 - f3B300 ), f3N200 ) * f3N200 ) ) / 3.0f;
    O.f3B201 = ( ( 2.0f * f3B003 ) + f3B300 - ( dot( ( f3B300 - f3B003 ), f3N002 ) * f3N002 ) ) / 3.0f;
    // Center control point
    float3 f3E = ( O.f3B210 + O.f3B120 + O.f3B021 + O.f3B012 + O.f3B102 + O.f3B201 ) / 6.0f;
    float3 f3V = ( f3B003 + f3B030 + f3B300 ) / 3.0f;
    O.f3B111 = f3E + ( ( f3E - f3V ) / 2.0f );
    
    // Compute the quadratic normal control points
    float fV12 = 2.0f * dot( f3B030 - f3B003, f3N002 + f3N020 ) / dot( f3B030 - f3B003, f3B030 - f3B003 );
    O.f3N110 = normalize( f3N002 + f3N020 - fV12 * ( f3B030 - f3B003 ) );
    float fV23 = 2.0f * dot( f3B300 - f3B030, f3N020 + f3N200 ) / dot( f3B300 - f3B030, f3B300 - f3B030 );
    O.f3N011 = normalize( f3N020 + f3N200 - fV23 * ( f3B300 - f3B030 ) );
    float fV31 = 2.0f * dot( f3B003 - f3B300, f3N200 + f3N002 ) / dot( f3B003 - f3B300, f3B003 - f3B300 );
    O.f3N101 = normalize( f3N200 + f3N002 - fV31 * ( f3B003 - f3B300 ) );
           
    return O;
}

Also my offer still counts, if you want my object space normal shaders (which don’t plan to be free for everyone :P)

i’m interested to know if you figured out the scaling issue? i’m also trying to do a few experiments and I’m really interested to know how this turned out. did you guys ever get displacement/height implemented into it as well?

[mysteryDate]

I have the same result as @Acey195 , despite the fact that my graphics card DOES support DOES support depth, ARGBFloat and render textures. As far as I know this should not be an NVidia vs ATi thing, but may be a Unity free vs. Unity pro thing (I know that is the case for RenderTextures at least). I’m running an NVidia GeForce GTX 970:

The Vertex-Hull-Domain-Fragment shader is rendering what looks like a single pixel, and when it is selected the wireframe is just one line:

I’m going to play with it a bit myself, but @RC-1290 , could you save me some time if you already know what’s going on? Thanks a million for your code.

[RC-1290]

@mysteryDate

As far as I know this should not be an NVidia vs ATi thing, but may be a Unity free vs. Unity pro thing (I know that is the case for RenderTextures at least).

Since you suspect it might be a Unity Free versus Unity Pro problem, have you tried it in Unity 5?

Actually, I just tested this on my new laptop, which uses a GeForce GTX 860M, and I get the same buggy results in Unity 5, as you describe, so I do think it’s a vendor specific problem.

[Edit] same problem in Unity 4 Pro, with Nvidia hardware. It might be worth looking at the alternative shader found above.

[OnikaDeeJay]

Hi everyone,

I know this thread is a bit old so sorry to bring it back to life however I’m struggling with this now and was wondering if anyone has got a good way of implimenting tessellation in a vert frag shader ??

Thank you all loads