Unity 4.2 - Stencils for portal rendering

[Noisecrime]

Update: 17.03.17
So its been a while, but finally got round to updating this project and putting the source on github for easier access and modification. The Project is available in versions for Unity 5.2.2.f1 and 5.4.2.f2.

View the Unity-StencilPortalRoomCube Project on Github.

Improved Shader Workflow.
Switched shaders to use Material Property Drawers which both make the stencil states modifyable per material and provides a nice ui in the inspector. This removes the need for many duplicate shaders and the old method of hard-coding values as well as streamlining the project.

Improved Project Layout
Cleaned up the entire project, folder structure, shaders, scripts, materials etc.

Support for Unity DepthNormalsTexture Generation
This means that image effects such as Edge Detection now work correctly with the stencil materials, though in Unity 5.2.2f1 it only works in dx9, the other api’s just fail - most likely a bug. Works fine in dx9, dx11 and openglCore in Unity 5.4.2f2.

Improved Example Scenes
The example scenes now have a UI that allow for enabling/disabling various options to observe their effects.

Additional Features

• Stencil Viewer
• Portal Culling
• Revamped Guide Document

Original Post

One of the more interesting features of Unity was the inclusion of access to the stencil buffer, which has a wide range of uses and enables a number of different effects. Two of the most popular being portals and stencil shadows.

In order to explore the new stencil feature I created a little pet project based around portal rendering, with a scene of 9 ‘room’ cubes, with each side of each cube showing a unique view into a different space.

If anyone has played Anti-chamber you’ll be familiar with the effect, but its a little hard to explain in words, so I made a quick video of my project to demonstrate the effect.

Unity 3D - Portal Room Cubes via Stencil Buffer

Its a cool little effect and it can have a variety of uses. I wouldn’t be surprised if Anti-chamber made extensive use of stencil portals for many of its mind-bending spatail effects (although its possible to achieve the same thing via Render To Texture portals too).

I’m intending to post the project source here as soon as i’ve cleaned it up and added some documentation. I’d hoped to provide a compressive guide to the effect and documentation, but time is against me, so i’ll likely just have to upload what I already have written.

It should be noted that use of the stencil buffer is only really practical in forward rendering as Unity uses most of it itself in deferred rendering. Its still possible to use it in deferred, but its probably not practical to do so.

Update 22.07.13 WebPlayer Available
Uploaded a webplayer demo which can be found here.

Update 23.07.13: Demo with full source for Unity 4.2
You can download the demo, full source unity project here.

Important!
To run the demo on mobile, you must go into the ‘Player Settings → Resolution Presentation’ tab and enable ‘Use 24 - bit depth buffer’. This will enable stencils on the device, otherwise the effect doesn’t work. However please note this project was designed for standalone (Win Mac) and the demo scene is not optimised for mobile, so performance is quite low.

[Fishypants]

Thats pretty awesome, great effect!

[fallingbrickwork]

Very cool!

[imaginaryhuman_1]

Neat.

Another thing you can do with the stencil buffer is to fill concave polygons without having to triangulate. I used it in another engine to fill dynamic bezier curves.

You can also use it for dynamic split-screen effects. For example in a two-player split, you can make the split at any angle by splitting along a line perpendicular to the line between the two players. Basically any shape you can render to the stencil buffer can act to stencil in/out the stuff you render to the color buffer. Like so:

I’m a bit disappointed this is a pro-only feature, because stencil buffers have been around for over a decade now and I kinda thought they’d stopped being supported because they were so old, better replaced with more modern methods such as render-to-texture. But I guess if Unity gave Free users the stencil buffer they could implement stencil shadows, and some other offects which normally need some kind of render-to-texture, and they don’t want that, right?

[SubZeroGaming_1]

Damn…That’s cool…

Make a tutorial on that.

[Noisecrime]

Thanks for the replies, should be uploading the source/tutorial soon.

@imaginaryhuman_1

Another thing you can do with the stencil buffer is to fill concave polygons without having to triangulate. I used it in another engine to fill dynamic bezier curves.

You can also use it for dynamic split-screen effects. For example in a two-player split, you can make the split at any angle by splitting along a line perpendicular to the line between the two players. Basically any shape you can render to the stencil buffer can act to stencil in/out the stuff you render to the color buffer.

Yeah there are quite a number of uses for it and its a nice alternative in many of them to using a render-to-texture system and thus save texture memory and improve quality (assuming you’re not doing 1:1 renderTexture to screen pixels). Fully expect to see stencil shadows and other effects come to the asset store in the next few weeks. Only wish I had the time to do some myself, but this little portal demo will have to suffice for now.

[Noisecrime]

Uploaded a webplayer of the demo and the full Unity project source.

Webplayer demo.

Demo, full source unity project.

[imaginaryhuman_1]

How are you doing the refractive glass, do you render the insides of the cube to a render texture and then pass it through a refraction shader?

[hippocoder]

Nice work. Will be interested to see shadow solutions for mobile pop up and that sort of thing. Can potentially be faster (and a hell of a lot better looking than low res shadow maps).

[kenshin1]

Really nice!

[Ziboo]

I just check your project, thanks a lot for your work.
There is even a doc !
Good job

[Dantus]

Awesome!

[Noisecrime]

@imaginaryhuman_1

How are you doing the refractive glass, do you render the insides of the cube to a render texture and then pass it through a refraction shader?

I’m just using the Unity Pro Glass shader (one of the default packages), which uses a grab pass of the entire screen. Obviously its not very accurate doing that, but artefacts aren’t really noticeable unless you look for them.

I did make one change to the Unity Glass shader as it used a grab pass per glass plane, which with 36 of them hammered performance. I changed the shader to use a ‘named’ grab pass, so all glass planes use the same one, which gave 3-4 times the performance. The glass shader also has its queue tag increased to ensure they are rendered and the grab pass is performed after rendering everything else. You’ll find all this commented in the glass shader.

@Ziboo

I just check your project, thanks a lot for your work.
There is even a doc !
Good job

Yeah I like to be thorough, though sadly I just don’t have the time to write the full documentation I was hoping to do. Hopefully its enough to give an idea about how to accomplish the effect and describe the problems and solutions.

Thanks to everyone else for your kind words. Hope the project is useful.

[icreate]

This is really awesome demo. Thank you for sharing.

I tried it quickly on my iPhone 5 and it hovers over 30fps but the stencil itself does not work.
i.e. nothing is being culled.
(I disabled the camera controls to get it to work with no major change to view a specific angle).

Is the stencil feature supposed to work on iOS?

The other interesting thing is that I noticed the anti-aliasing was turned on in the quality settings. I am no expert but when I turned it off, the fps dropped to 10fps. I did not really expect that. am I missing something?

[hippocoder]

Stencil afaik does work on iOS. If it isn’t working for you, perhaps a bug report is in order.

[Noisecrime]

@icreate

This is really awesome demo. Thank you for sharing.

I tried it quickly on my iPhone 5 and it hovers over 30fps but the stencil itself does not work.
i.e. nothing is being culled.
(I disabled the camera controls to get it to work with no major change to view a specific angle).

Is the stencil feature supposed to work on iOS?

The other interesting thing is that I noticed the anti-aliasing was turned on in the quality settings. I am no expert but when I turned it off, the fps dropped to 10fps. I did not really expect that. am I missing something?

Hadn’t even thought to test it out on iOS and my first thought would be that its not supported, but I don’t remember reading that anywhere. Could maybe be an issue with the glass shader, so i’d probably try removing some of those first and rebuilding to see what happens. Its odd really as although the project construction is somewhat complex, the actual features are not, so off the top of my head I can’t think of a good reason for this, even less so the problem with AA.

I’ll break out the mac and iPad and have a quick look.

I should point out though the sample scene was never designed for mobile, although fundamentally the effect here should work with the stencil shader, the actual scene is loaded with point lights, fancy screen effects (glass shader), forced anisotropic filtering and AA etc. So whilst it will be useful to ensure it does work on iOS, it is not a scene I would recommend for iOS or mobile in general.

[Noisecrime]

@icreate

This is really awesome demo. Thank you for sharing.

I tried it quickly on my iPhone 5 and it hovers over 30fps but the stencil itself does not work.
i.e. nothing is being culled.

Thanks to the awesome developers at Unity Technologies, this issue is now resolved.

Its not a bug, just a simple case that in order to use stencils on iOS devices (possibly all mobile devices?) you need to go into the ‘Player Settings → Resolution Presentation’ tab and enable ‘Use 24 - bit depth buffer’. Doing this will enable the stencil buffer on these devices and I can confirm that the demo project now runs correctly.

However it also runs really slowly (8 fps on iPad 2), though that is not unexpected as the scene was designed on and for desktops, it is heavy in pixel lights ( 1 point light per room and a directional light), has a large number of gameObjects, fancy glass effects using grab pass etc. So although i’ve not tested, I doubt using stencils is too much of a performance drain, far more likely to just be the unoptimised scene i’m using.

[icreate]

@Noisecrime

‘Use 24 - bit depth buffer’…

Thank you very much, it works on my iPhone now.

I can also confirm that turning off Anti-aliasing stops stencil from working and makes the performance worse.

I removed all the point lightings an changed all the bumped specular shaders to the mobile version. Yes it is not as a nice but it made it work at 60fps on my iPhone5, but if I turn off the anti-aliasing, stencil stops working and the performance drops to 14fps.

Any idea what is the link between stencil and anti-aliasing?

[Noisecrime]

@icreate

Thank you very much, it works on my iPhone now.

Any idea what is the link between stencil and anti-aliasing?

No idea, pretty sure in my tests I’d disabled AA early on and everything worked fine. Can’t comment on performance changes with AA off, though it sounds very counter-intuitive.

In terms of optimsiations there is much that should be done with the scene. For example Unity has no notion of portals (at least not stencil portals as used here), so even though you might only see the contents of 1 or 2 portals in the same cube room, its still rendering all four portal contents, just that most are being culled by the stencil.

It would probably be advantageous then to add some code to determine which portals are facing the camera, and hide/show or enable/disable the contents within them. This would drastically cut down the number of objects trying to be rendered.

[icreate]

I liked the demo so I spent some time to find where exactly the performance issue is.

The glass shader (I also tried the original version) is gaining performance when AA is on. Any Unity developer can help explaining this?

I can also confirm that your version of the glass shader is definitely faster. Nice job there.