Currently when I use a stencil buffer the shadows still cast even though the object is invisible. Is there a way around this? I’ve tried using a custom shadow pass with the same stencil buffer code but it still writes everything to the shadow map buffer. I’ve tried also using SV_Depth but it seems like the stencil buffer doesn’t seem to work in the Shadows.RenderJob
Thanks
The object is invisible … from the point of view of the camera. It’s likely still totally visible from the view of the light. Plus stencil stuff tends to not work great in shadows as Unity doesn’t respect render queue orders.
Stencils are a 2D effect. Hiding an object on screen is hiding them for those pixels that the geometry overlaps for that view. If the geometry doesn’t overlap in another view, then it’ll be rendered for that view.
Options:
Don’t cast shadows for objects that are hidden.
Don’t use stencils, instead use a clip() shader on hidden geometry and some kind of projected shape. This a lot more work, but will be consistent.
Use two cameras and render textures like a lot of portal tutorials out there show.
Thanks for the info I was afraid of that.
Does that mean when something gets rendered into the hardware Depth buffer the stencil value gets completely ignored even if set correctly (I use custom shadowcaster passes)?
Stencils and depth buffers work just fine together. That’s part of how your object is disappearing to begin with. The problem is the render order is very important when dealing with stencils. You want anything the stencil isn’t going to affect to render first, then the stencil, then anything that it needs to affect after that.
The problem is two fold for shadow maps. One being that Unity ignores the render queue and sorts objects front to back no matter what, but that can be managed by rendering your objects into the shadow maps manually with a command buffer instead of letting Unity do it automatically. The harder part is the fact that just because something is being masked from your main camera doesn’t mean the light can’t see it.
(Sh*tty diagram incoming)
Possible SOLUTION: For anyone who bumps into this post and is having trouble with shadows on their invisible objects. I fiddled around while using stencil shaders to setup a portal effect and got something pretty decent as a workaround. Here are the steps taken below.
2)On first directional light in scene go to its component and change to ‘baked’ (must do this first before next steps OR process will not work correctly!)
3)Go to Unity ‘Light Settings’ and bake your lightmap. Afterwards you should see that shadows have baked ONLY onto items marked as static. This will make your invisible items lose their shadow entirely also. We still likely want to see shadows on those object when the object itself is not invisible intentionally… we’ll fix this in the next few steps.
4)Overlap other area (if your going for a portal effect) or duplicate current area on top of existing one (if you just want shadows to appear when viewed through the plane you are using as a stencil filter). Note: Be sure that everything in the newly overlapping area, which you do want to have a shadow, is using your shader which allows objects to be viewed when seen through the stencil filter plane. So basically as you would normally set this sort of thing up anyway, ie one shader on object to be seen and different shader on object to do the seeing. Also KEEP this environment as NOT STATIC.
5)Now, add second directional light to scene and keep this set as ‘realtime’. You will notice that now a light version of a shadow is appearing on invisible objects yet again, this is momentary. We have added this second light so that we can have shadow on objects when they are intentionally viewed.
6)Now, select first environment (all objects which were marked static) and create and add them to a new layer. I called mine ‘No Shadow’ just for easy use.
Hopefully this will help others… seemed to work for me 
so i cannot using dynamic direct light effect by using this solution? sorry i m a nooooob
thanks
