I need a more complicated “Hole in wall” effect; the hole should only hide objects that are between it and the camera, objects behind the hole have to be rendered. The Object with the hole in it also has to be rendered in the area that isn´t covered by the hole.
I basically want a space (for example a cylinder) where nothing is rendered, but everything outside this space has to be rendered normally.
All solutions i tried before hide everything that the hole in the wall covers, also geometry that belongs to the same game objects but is behind the hole.
Explanation to the picture: Everything green should be rendered, the red things should not be rendered. The red circle in green square represents the hole in the wall. The rest of the room must be rendered.
You may want to look at using stencils, but these come with a whole host of limitations.
The basic idea is you have your red object render to the stencil buffer, with a very low material queue, and [using ICODE]ColorMask 0 ZWrite Off[/ICODE] so that it is rendered first but invisible and not to the depth buffer. Then you render your green wall that you want to have get “cut out” by the wall use stencil settings that exclude it from rendering where the red object already has.
Here are some of the issues you’ll face:
It won’t matter where the red object is in terms of distance from the camera. Any screen pixels it’s rendered in the green wall won’t appear at. Stencils are purely 2D, so you can’t have the wall have a hole in it only when the red object is closer to the camera than the wall.
Unity’s depth textures such as the camera depth texture, used for receiving directional shadows and post processing effects like depth of field, and shadow maps don’t respect material queue order, so there’s no guarantee the red object will render first. So the hole that appears in the main rendering may not be taken into account in other effects. That can be worked around by using command buffers to force the object to be rendered in the order and render passes you want.
If by chance you want to use deferred rendering, you may be completely out of luck as stencils are disabled for deferred rendering (most of the time, again may be possible to work around with command buffers).
Other options you have vs using stencils:
Render your object to a render texture from the view of the camera and have a custom shader on your object that you want to punch a hole in that samples that render texture. Similar to stencils, but a bit more manual. One advantage is you could render your object as a depth and use that to only put a hole in the wall if the wall is further from the camera than your object.
Use analytical shapes instead of meshes. Using a custom shader you could raytrace an analytical shape to create a mask for the wall in the wall’s shader. You can even do depth comparisons like the render texture approach since you’ll know the object’s depth from doing the raytracing.
Use render textures and a second camera that renders the scene without the wall. Display the render texture on a mesh using screen space UVs. Probably the most expensive option, but one of the most straightforward setups.
Thank you for your reply!
I didn´t want to use Stencil Shaders (mostly because of your first point about them) and ended up writing a custom Shader that discards all Pixels in a certain cuboid. The cuboid is “built” in a script and the corner points are given as Vectors to the material. I wrote this in an URP Project, but it should theoretically work in a normal 3D project, too.
However, when i implement the Shader in a 3D project, the material doesn´t get updatet in Play Mode? I can see in the Inspector window that the correct corner points are transferred from the script to the shader, but the hole in wall effect happens only after leaving play mode (at the correct place of the wall, so it basically works).
Here is how i transfer the corner points:
