Is it possible to smooth stencil mask edges (Sight of view) ?

[MichalDev]

Hello dear developers. This post is follow up to my previous follow up which you can find here : https://forum.unity.com/threads/character-field-of-view-visualisation-shader-grayscale.523226/ . I solved the problem in a rather peculiar way. I’ve created another camera for overlay layer which render everything else than MainCamera which renders only player’s vision mask. I used only depth option on main camera and tweaked depth options. I got what I wanted. But there is another problem.

As you can see vision mesh edges are rather sharp. Is there any way to smooth it a little bit ? As I know stencil buffer can only render or not render pixels. Additional information is that I’m rendering this mesh programatically, maybe I need to add some smoothing when rendering ?

I will be eternally grateful if someone help me to overcome this problem !

[Zee_pso]

Do you mean the edges around the view circle? I’m assume the circle is a static portion of the mesh, which opens out towards the player’s forward direction. If so, just add a few more vertices to create a smoother transition when you’re building it to round it out.

If you mean just in general, I’d check out this tutorial.

[MichalDev]

Yes, both circular shape around player and conical shape in front of player are meshes generated in script. But I don’t think so I can smooth them adding extra pixels because it’s still stencil buffer and pixels won’t be rendered at all. Correct me if I’m thinking wrong

[MichalDev]

His mesh edges all also sharp but in his case it looks good because of totally black background. I have grayscaled world outside the mesh so for me this sharp transition looks bad.

[daxiongmao]

you could possibly think about writing your mask to a screen space texture instead.

then you could do a blur as a preprocess then use it as a mask. or when generating it you can add smooth ramps to the edges.

then apply it either in a post process pass. or sample it during rendering of world.

you will have to evaluate its performance.

using the stencil now are you having to draw your world twice? once for grayscale and once for color?

[MichalDev]

@daxiongmao

you could possibly think about writing your mask to a screen space texture instead.

then you could do a blur as a preprocess then use it as a mask. or when generating it you can add smooth ramps to the edges.

then apply it either in a post process pass. or sample it during rendering of world.

you will have to evaluate its performance.

using the stencil now are you having to draw your world twice? once for grayscale and once for color?

I’ll post you my current way of doing it later, because now I don’t have access to Unity.

[MichalDev]

For now i have hierarchy like that:

First is VisionCam. This cam renders only field of view and it is set as depth only.

Between VisionCam and OverlayCam I have plane with Stencil Mask shader on it:

If I left at this level I would achieve something like that :

But I wanted everything else to render normally but in grayscale and some other effects. So I’ve added next camera:

Maybe you can think of another, easier approach. I also want to add that I’m not supposed to render any enemies beyond field of view so I got desired effect with overlaycam culling mask settings. Thanks

[Zee_pso]

@MichalDev

Yes, both circular shape around player and conical shape in front of player are meshes generated in script. But I don’t think so I can smooth them adding extra pixels because it’s still stencil buffer and pixels won’t be rendered at all. Correct me if I’m thinking wrong

Ah, I thought you meant sharp edges as in the angles of the fog. Do you mean a transition effect? Un-Blurring the view as it gets towards the view cone?

You could make a second mesh that wraps along the edges of the view mesh, and apply a grab pass shader to blur what’s under it to a lesser extent, this will give you a smoother transition. Alternatively, you could play with the alpha values of the mesh’s vertex colors and make the vertices towards the inside have an alpha of 0, and towards the outside 1, and fade between the fog and screen camera, but the grab pass would probably be easier to set up.
Example:

As for the camera question, you don’t need a second camera if you inverse the view mesh, so it cuts a hole where you can see, and use a grab pass to blur what’s under the mesh and a stencil to obscure, but a two camera set up probably makes more sense and would be easier to work with.

[daxiongmao]

I am still not clear exactly on what is drawing what.
Your visioncam you say only draws the field of view but its culling mask is set to everything. Is this drawing your FOV mesh and writing to the stencil buffer?

Also when you say depth only, Are you talking about the clear flags? This is just clearing depth. It is still drawing everything.

Hmm, it seem to me you maybe drawing everything 2x. I can’t see you exact culling but the first camera draws everything. The second camera is mixed so i am not sure what is there.

The plane has the stencil shader is that just writing black where it passes the stencil test?

What is your platform?

I don’t think you are going to be able to get smooth with the stencil. When i think smooth i am thinking that you just want to fade to black. Not perform some kind of blur etc.

I don’t know your performance limitations but I would probably create your fov mesh, then expand it a little bit. And in the expansion area i would use some vertex data to blend between 1 and 0. You could then render this to a 8bit render texture. This should give you a nice black / white image with a transition around the edges.

Then you can just render your scene normally. But use this texture as your mask when drawing. In your environment shader you can look it up and based on what it reads you can lerp between the full color and maybe grayscale/luminance of that color.

Then for your enemies shader you can do something similar but alpha blend them out instead or alpha test/kill them.

This will require an extra texture read but if you lower the resolution of the render texture it may not be too much overhead.

1. Render special mask with smoothed borders to render texture. (Shader)
   If its too hard to inset/outset your mesh, you could blur this texture.
2. Set render texture to global shader / materials
3. In environment shaders sample render texture mask to output color or grayscale
4. In enemy shaders sample render texture mask to blend out enemies.

A second idea could be to possibly dither the stencil in those areas. not sure exactly how to do this but when rendering your mesh you determine a dither pattern based on the transition location.

Sorry if its not clear or doesn’t fit your case.

[MichalDev]

@daxiongmao

I am still not clear exactly on what is drawing what.
Your visioncam you say only draws the field of view but its culling mask is set to everything. Is this drawing your FOV mesh and writing to the stencil buffer?

Also when you say depth only, Are you talking about the clear flags? This is just clearing depth. It is still drawing everything.

Hmm, it seem to me you maybe drawing everything 2x. I can’t see you exact culling but the first camera draws everything. The second camera is mixed so i am not sure what is there.

The plane has the stencil shader is that just writing black where it passes the stencil test?

What is your platform?

I don’t think you are going to be able to get smooth with the stencil. When i think smooth i am thinking that you just want to fade to black. Not perform some kind of blur etc.

I don’t know your performance limitations but I would probably create your fov mesh, then expand it a little bit. And in the expansion area i would use some vertex data to blend between 1 and 0. You could then render this to a 8bit render texture. This should give you a nice black / white image with a transition around the edges.

Then you can just render your scene normally. But use this texture as your mask when drawing. In your environment shader you can look it up and based on what it reads you can lerp between the full color and maybe grayscale/luminance of that color.

Then for your enemies shader you can do something similar but alpha blend them out instead or alpha test/kill them.

This will require an extra texture read but if you lower the resolution of the render texture it may not be too much overhead.

1. Render special mask with smoothed borders to render texture. (Shader)
   If its too hard to inset/outset your mesh, you could blur this texture.
2. Set render texture to global shader / materials
3. In environment shaders sample render texture mask to output color or grayscale
4. In enemy shaders sample render texture mask to blend out enemies.

A second idea could be to possibly dither the stencil in those areas. not sure exactly how to do this but when rendering your mesh you determine a dither pattern based on the transition location.

Sorry if its not clear or doesn’t fit your case.

You helped me a lot man. I don’t understand everything but I will try to implement your point of view in game. The platform is Windows so I dont have as many limitations as on android. I’m not good at writing shaders so it can be a little bit hard for me but I will try

[MichalDev]

But if view mask is mesh generated in runtime how I can render it to the texture ?

[daxiongmao]

You just assign the camera rendering your mesh a render texture. I usually do this through code but you can also create one in the editor. You would then set it the cameras target texture. You then can also set it as a texture on a material.

Set your generated model to a specific layer and then have this camera cull setup only for this layer.
Make sure it renders first in the by setting the depth. I am asssuming you can already do this? Because you are rendering the stencil.

I would probably create the render texture in code. That way you can adjust its resolution based on screen size.
You probably only need .5 or even .25 of screen size.
You can just use a single r8/a8 format and not need depth.

Then I believe you should be able to do something like shader.settexture and assign it globally.
That way it will be available to your material shaders.
You can add the correct named sampler and then read from it in your other shaders.

Sorry I don’t have specific functions and type names. I’m on my iPad.

[MichalDev]

@daxiongmao

You just assign the camera rendering your mesh a render texture. I usually do this through code but you can also create one in the editor. You would then set it the cameras target texture. You then can also set it as a texture on a material.

Set your generated model to a specific layer and then have this camera cull setup only for this layer.
Make sure it renders first in the by setting the depth. I am asssuming you can already do this? Because you are rendering the stencil.

I would probably create the render texture in code. That way you can adjust its resolution based on screen size.
You probably only need .5 or even .25 of screen size.
You can just use a single r8/a8 format and not need depth.

Then I believe you should be able to do something like shader.settexture and assign it globally.
That way it will be available to your material shaders.
You can add the correct named sampler and then read from it in your other shaders.

Sorry I don’t have specific functions and type names. I’m on my iPad.

Thanks, you really helped me a lot ! Now I need to find some time to implement this. I hope to get a desired effect like that :

[MichalDev]

I am currently stuck at making shader that will use this texture … I was learning shaders all day without effects. I remembered that I used to have shader that was using light to fade out characters. It was really great and smooth, but I can’t use light for field of view but I want this “smooth feeling”. If I only can add some alpha mask to the target texture or something like that : [Solved] How to add an alpha mask over the existing Transparent/Cutout/Soft Edge Unlit shader ?

[daxiongmao]

were you able to render your fov mesh to the render texture?

[MichalDev]

Yes. It’s working as intended.

Here’s a piece of code responsible for creating render texture :

Attached to view camera:

But now I need to mask this global texture in shader somehow and it’s still won’t solve my problem with smooth edges. I hope there is a way to overcome this problem. I can create smooth circular mask for field of view in shader but I don’t know if it’s possible with the triangular mesh I’ve created. I’m struggling with that for a long time.

[daxiongmao]

Cool.
I setup some example stuff for you.

Do you think you could create an outset or inset area for your view mesh?

I made this in maya but if you could do something like this when you generate the mesh you can adjust your edge blends.

There are two shaders i made for rendering the mask RenderMask, RenderMaskHard.
The mesh could use vertex color or uv coordinates. My model has configured both but the shader uses the UV.
In the model the outer verts are all UV (0,0) and the inner verts are all (1,1).
In the scene there are two models SampleView and SampleViewHard you can toggle them to see the difference.

To render the mesh I created another shader UseMask. It reads the mask (Screen Space) and then uses the value to blend between the original color and the luminance value of that color.

void surf (Input IN, inout SurfaceOutput o) {
    fixed4 c = tex2D(_MainTex, IN.uv_MainTex) * _Color;
   
    float2 screenUV = IN.screenPos.xy / IN.screenPos.w;
    fixed m = tex2D(_ViewMask, screenUV);

    o.Albedo = lerp(c.rgb, Luminance(c.rgb), m);
    o.Alpha = c.a;
}

The last thing would be if its too hard to generate this kind of mesh you can just apply a postprocess effect to the camera rendering the mask and apply a blur to it.

3444673–272662–TestView.zip (32.2 KB)

[MichalDev]

@daxiongmao

Cool.
I setup some example stuff for you.

Do you think you could create an outset or inset area for your view mesh?

I made this in maya but if you could do something like this when you generate the mesh you can adjust your edge blends.

There are two shaders i made for rendering the mask RenderMask, RenderMaskHard.
The mesh could use vertex color or uv coordinates. My model has configured both but the shader uses the UV.
In the model the outer verts are all UV (0,0) and the inner verts are all (1,1).
In the scene there are two models SampleView and SampleViewHard you can toggle them to see the difference.

To render the mesh I created another shader UseMask. It reads the mask (Screen Space) and then uses the value to blend between the original color and the luminance value of that color.

void surf (Input IN, inout SurfaceOutput o) {
    fixed4 c = tex2D(_MainTex, IN.uv_MainTex) * _Color;
  
    float2 screenUV = IN.screenPos.xy / IN.screenPos.w;
    fixed m = tex2D(_ViewMask, screenUV);

    o.Albedo = lerp(c.rgb, Luminance(c.rgb), m);
    o.Alpha = c.a;
}

The last thing would be if its too hard to generate this kind of mesh you can just apply a postprocess effect to the camera rendering the mask and apply a blur to it.

Thanks a lot man. I will try this out as soon as possible !

[MichalDev]

Hey, it’s working like a charm ! Definately better approach than mine. Still I have to figure mesh insetting and smoothing thing out. But for now I want to ask you a question.

Can you explain me the yellowish part ? Why did you do that ? I also wanted to render everything else in greyscale. Not the viewmesh itself. Thank you a lot for your help. I really appreciate that.

[MichalDev]

I was happy prematurely. On your scene everything works great but when I attached my mesh something went wrong.