I am trying to use a vertex displacement shader which moves the vertices of an object in the x-z plane. I also have an edge detection effect on the camera (using Roberts Cross Depths Normals).
The problem is that the edge detection seems to ignore the vertex displacement created by the shader and draws an edge where the edges of the objects were originally. I have tried to change the Queue tag of the shader but that didn’t help. It’s not a problem with the vertex displacement shader - I’ve also tried several other vertex shaders (which I didn’t write myself), but the same thing happens with the edge detection.
It seems to me that the edge detection is happening before the vertex deformation, but I couldn’t find any means of controlling the order. Could anybody please help?
does your edge detection use a depth buffer created in a separate pass?
The edge detection is the standard unity edge detect normals image effect. I don’t think they use a separate pass for the depth buffer.
ok, so your depth buffer will be created with a built in (“hidden” ) depth and normals shader, which doesn’t use your displacement. Hopefully, there is a way to edit that shader, or just for your displacement objects, so that the depth and normals pass uses your displacement:
http://docs.unity3d.com/Manual/SL-CameraDepthTexture.html
http://docs.unity3d.com/Manual/SL-ShaderReplacement.html
Deferred or forward rendering?
Surface shader or vert/frag?
I have been using forward rendering and the model remained undeformed for the edge detection image effect. However it works well for the deferred rendering. I’ve used both surface and vert/frag shaders with the same result. Obviously for vertex displacement I’ve used a vertex program even in the surface shader.
Is there any way I can achieve the same result in the forward rendering path?
The depth comes from the ShadowCaster pass. With a surface shader you can add “addshadow” to the #pragma surface line and it’ll generate one automatically. Otherwise you need to write one yourself otherwise it’ll use the fallback shader’s shadow caster pass which does not have any of your vertex modifications.
For the normals when using the forward render path you need to download the builtin shaders and copy Camera-DepthNormalTexture.shader into a resources folder in your project and add a new subshader with a unique RenderType tag that does the vertex modification. Then set that RenderType on your own shaders.
Click on download → built in shaders
In deferred, assuming your shader has a deferred pass, the normals get rendered out into the gbuffer at the same time as everything else so the normal just work.
I’ve already tried the addshadow option. It does work when the edge detection is set to Sobel Depth or Sobel Depth Thin, but it still doesn’t work for Roberts Cross Normals. The thing is, the even though I’m deforming my vertices, in my particular case I’m not affecting normals. In my game, I’m deforming the outer borders of a flat tile in its plane, so the normals are always pointing upwards, perpendicular to the plane. I’ve confirmed this by using a shader showing the normals as a colour. Both with and without displacement, the normals show as green, as expected.
Because addshadow fixes up the depth, which is what the Sobel outlines use. Roberts Cross Normals uses the same depth, plus the normals. However if the normals and depth don’t match up you’ll get weird stuff which is what you’re probably seeing.
Btw, using the Windows > Frame Debugger you can see what the depth and normal passes look like.
Thanks. In the Frame Debugger it shows that the depth buffer is being updated, but not the normals. I’ll look into the solution you provided for forward rendering - not sure I understood exactly how to do it, but I’ll try 
Interestingly enough, in deferred rendering, although the Roberts Cross Normals mode works for a perspective camera, it doesn’t for an orthographic camera - depth is updated properly but normals aren’t (even though the inspector still says that the camera mode is set to render both depth and depthnormals).