Screen Space Local Reflection

Hi all.

I have banged my head against all the different “spaces” in Unity3D projections maths stuff and gave a go with what people call “Screen Space Local Reflection”

The idea is certainly not new, and some people here in Unity community has already tried them and I know for sure that the hard surface shader has it working really nicely, but there is nothing like doing it yourself.

So here it is…

As you can see from the picture, that I have basic stuff going… but there are still lots of improvements needs to be done.

The one particular improvement that I am looking for is :

When you have wrong ( or no info ) to reflect because the marched ray hits under the model or back of the model, how do you mask them and fade them away from the reflection contribution?

Any comments or ideas especially from the ones who has already done this before would be more than welcome.

My method is basically based on this post.

http://www.gamedev.net/blog/1323/entry-2254101-real-time-local-reflections/

The above method also suffers from the same problems…

Another thing is to find some better method of ray marching…

Thanks!

I have tried to make this shader as a post process shader…

But I am having trouble converting viewPos into screenPos.

Basically, I need to convert view space position into screen space position using projecting matrix, but I can’t make it work for the post processing shader.

It works for non-post process shader.

Please note that this is an experiment shader code.


Shader "Custom/SSRPostProcess" 
{

    Properties 
    {
		_MainTex ("Base (RGB)", any) = "" {}
    }
           
    SubShader 
    {
        Pass 
        {
            CGPROGRAM
			#pragma exclude_renderers d3d11 xbox360
            #pragma vertex vert
            #pragma fragment frag
            #include "UnityCG.cginc"
            #pragma target 3.0
            #pragma glsl
            
            sampler2D _MainTex;
            float4 _MainTex_ST;
            float4 _MainTex_TexelSize;
           
            
            uniform sampler2D _backgroundTexture;
            uniform float _fadeToView;
            uniform float4x4 _inverseProjection;
            uniform float4x4 _ProjMatrix;
            uniform float4x4 _ProjectionInv;
            uniform float4 _ProjInfo;
            
            sampler2D _CameraNormalsTexture;
            sampler2D _CameraDepthNormalsTexture;
            float4 _CameraDepthNormalsTexture_ST;
            
            sampler2D _CameraDepthTexture;
            float4 _CameraDepthTexture_ST;

            struct v2f {
                float4 pos : SV_POSITION;
                float2 uv : TEXCOORD0;
            };

            v2f vert (appdata_img v)
            {
                v2f o;
                o.pos = mul (UNITY_MATRIX_MVP, v.vertex);
               	o.uv =  v.texcoord.xy;
				return o;
            }
            
            float3 ReconstructCSPosition(float2 S, float z) 
			{
				float linEyeZ = LinearEyeDepth(z);
				return float3(( ( S.xy * _MainTex_TexelSize.zw) * _ProjInfo.xy + _ProjInfo.zw) * linEyeZ, linEyeZ);
			}
			
			float4 ReconstructViewPosition(float2 S, float z) 
			{
				// for reference
				float4 clipPos = float4(S*2.0-1.0, (z*2-1), 1);
				float4 viewPos;
				viewPos.x = dot((float4)_ProjectionInv[0], clipPos);
				viewPos.y = dot((float4)_ProjectionInv[1], clipPos);
				viewPos.w = dot((float4)_ProjectionInv[3], clipPos);
				viewPos.z = -z;
				viewPos = viewPos/viewPos.w;
				return viewPos;
			}

             
            half4 frag (v2f i) : COLOR
            {
            	float4 geom = tex2D(_CameraDepthNormalsTexture,i.uv);
            	float3 viewNormal;
            	float  depth;
            	
            	// ths is to get a view normal
            	DecodeDepthNormal(geom,depth,viewNormal);
            	
            	// just get Z again from depthmap because I don't want linear Z from 0-1 initially..(I think)
            	float d = ( tex2D(_CameraDepthTexture, i.uv.xy) );
            	float3 scrPos = float3(i.uv * 2 -1,d);
            	
            	float4 viewPos = ReconstructViewPosition(i.uv,d);
            	float3 vspReflect = normalize(reflect(normalize(viewPos.xyz), normalize(viewNormal)));
            	
            	
            	// I think the problem is here
            	//float4 vspPosReflectT = mul (UNITY_MATRIX_P,viewPos+float4(vspReflect,1));
            	float4 vspPosReflectT = mul (_ProjMatrix, viewPos +float4(vspReflect,1));
   				float3 vspPosReflect = vspPosReflectT.xyz / vspPosReflectT.w;
            	
            	            	
            	// subtract to get a vector.. (sub from end to start)
            	float3 sspReflect = normalize( vspPosReflect - scrPos );
            	
            	sspReflect = float3(sspReflect.xy*0.5f,sspReflect.z);
            	
            	//initial stepping scalse value.. not sure about this neither
            	float stepScale = 2/_ScreenParams.x/length(sspReflect.xy);
                stepScale = 0.002;
                
                float4 c;
                float sampleDepth;
                float currentDepth;
                float depthTest = 0; 
                float len = 0;
                int maxCount =164;
                float4 r = 0;  
                float bias = 0.001;
                
                float rcpfadefact = 1/(1.0 - _fadeToView);
   				float faceviewerfactor = (vspReflect.z - _fadeToView) * rcpfadefact;
                faceviewerfactor = 1-faceviewerfactor; 
                
                float3 startPosSS = float3(i.uv,scrPos.z);
                float3 sspReflectFinal = sspReflect;
                sspReflectFinal  *= stepScale * 1;
                
                c = tex2D (_MainTex,i.uv);  
                
				float deltaD = 0;
				float3 samplePos = startPosSS + sspReflectFinal;
				float3 oldPos = startPosSS;
				int cnt=1;
				
				// ray march
				
				while (cnt < maxCount)
                {
                	 
                    geom = tex2Dlod (_CameraDepthNormalsTexture, float4(samplePos.xy,0,0));
                    sampleDepth  = DecodeFloatRG(geom.zw);
                    currentDepth = Linear01Depth(samplePos.z);
                    
        			if ( sampleDepth < currentDepth)  
                    {  
                    
                    	deltaD = (currentDepth - sampleDepth);
                    	
						if (deltaD < bias)
	               		{
	               			depthTest = 1;
	               			cnt = maxCount+1;
	               			break;
	               		}
	              		else
	                	{
	                		sspReflectFinal *= 0.5;
							samplePos = oldPos + sspReflectFinal;
	                	}						
                   	}
                   	else
                   	{
                   		oldPos = samplePos;
                   		sspReflectFinal *= 1.1;
						samplePos += sspReflectFinal;
                   	}
                   	len += 1;
                   	cnt += 1;
               	} 
                
               float3 endP = float3(samplePos.x*2-1 , samplePos.y*2-1,samplePos.z);
               
               float dist = abs((distance(scrPos.xyz,endP)));
               
               r = tex2D(_MainTex,samplePos.xy);
               r *= faceviewerfactor* faceviewerfactor;
               r *= (1- pow(dist,1));
               r *= depthTest;
               
               return  c + saturate(r)*0.5f ;                
             	
            }
            ENDCG
        }
    }
}

I think this line here is the problem… (not 100% sure but)

float4 vspPosReflectT = mul (UNITY_MATRIX_P,viewPos+float4(vspReflect,1));

This works for non post process shader, but for some reason doing this and then doing the next line

float3 vspPosReflect = vspPosReflectT.xyz / vspPosReflectT.w;

doesn’t give me the correct result. For some reason vspPosReflectT.w part is wrong…

Any ideas?

http://forum.unity3d.com/threads/178307-My-SSR
the raw stuff did by me:)

Hi Lucifer

I think I saw that one some time ago. very nice.

Tell me did you try doing that as a post processing effect?

there is something was missed in my implementation,but i didnt figure out how to fix or improve that,before that i wont try post effect
after all it is something few months ago,never have time to try more :slight_smile:

looks nice)

Ok, after yet another head banging against world / view / projection space stuff I got this to work using post processing shader.

My main struggle was to convert the screen space positions into view space positions and then back to the screen space. Also had hard time converting world space normal into view space normal.

I could not use the view normals from Unity provided view+depth texture. Because view normals were too inaccurate and caused jump when I turn camera around. I had to land up using _CameraNormalsTexture and convert it into view space normal instead. But this ment I had to use deferred shadering. :frowning: oh well…

Performance wise, I think it must be slow or at least could go much faster. I couldn’t really tell because fps tells me I am getting crazy 4000 fps… but I know that can’t be true… looking at the profiler tells me that my post process is taking around 3ms so maybe it is around 200fps or something…?

It is still the same basic SSLR without any advanced optimizations or visual improvements either than fading off on the edge of screen and obvious reflection vector pointing towards view…


I think I need to come up with some better ray marching techniques and need to improve on depth compare algorithm…
Also thinking about sampling and ray marching on smaller reduced texture to reduce the number of ray marching steps… I have seen from the other posts that their steps are like 16-30 etc… but my steps are like 40 so I am sure there must be some better ways.
Simple blur may help too…
There are just so many things to learn…

Any comments or feed back welcome. I would like to very much share and learn from others who have attempted this technique.

good work. How are you computing _ProjMatrix in c#, why are you not using UNITY_MATRIX_P, can you please post your code or PM me please…cheers

Here is pic of my progress so far.

I have added some blur pass after the reflection pass to reduce the artifacts from the false reflection catch and z depth in accuracy from ray marching. This one is combination of cube map reflection used for the sky.

To calculate projection matrix from C# :

		P  = this.camera.projectionMatrix;

		bool d3d = SystemInfo.graphicsDeviceVersion.IndexOf("Direct3D") > -1;
		if (d3d) {
		    // Scale and bias from OpenGL -> D3D depth range
		    for (int i = 0; i < 4; i++) {
		        P[2,i] = P[2,i]*0.5f + P[3,i]*0.5f;
		    }
		}

and pass it using Shader.SetGlobalMatrix

The reason why I am calculating it instead of using UNITY_MATRIX_P is that in image effect (post processing) UNITY_MATRIX_P becomes an identity matrix… I spent so long trying to figure that one out. When you know it , it kinda makes sense but it was not useful for me and confusing as hell. I don’t think this is documented anywhere in the official Unity doc.

What a sexy shader.

Nice work Castor! Looks promising. :slight_smile:

Thanks Save…

I am trying to figure out better method for doing ray marching but more I dig into it more I think I am approaching this realm of super maths.
(Should have paid more attention at school! :smile: )

There is this “distance field” method which I understand the principles but I don’t think it can just be used in this case. Depth map per frame is not some mathematical formula. And calculating 3d distance field map sound like too expensive for realtime.

Yes that UNITY_MATRIX_P issue is weird, any other tricky things to look out for? are you still using CameraDepthNormalsTexture to read depth? im also trying to implement this to learn from…cheers again

No, I am using :

depth = UNITY_SAMPLE_DEPTH(tex2Dlod (_CameraDepthTexture, float4(samplePos.xy,0,0)));

which is non linear depth. It is the same as .z of the screen position z (from vertex shader if not using post process)
Thing is that because it has very good accuracy around near plane of camera, I convert it into Linear01Depth when I do the compare.
But I don’t know if this is the best way because it just means more instructions per ray march loop… :frowning:

Another thing to note is that view normal from CameraDepthNormalsTexture is almost free because Unity renders anyway, but its accuracy is not really good when it comes to the long flat plane. I think it is due to the compressing it into 2 RG channel and then decoding it back… The issue is that because it is not accurate enough, your refilection vector “jumps” in steps for large flat planes. ( I guess the error just propagates and multiply when you calculate reflection) so I could not really use them.

What I landed up is going to deferred rendering path and used _CameraNormalsTexture instead. That gives you world normal so I had to convert it into view normal. (Again using projection from C#)

Matrix4x4 V = this.camera.worldToCameraMatrix;
Shader.SetGlobalMatrix ("_ViewMatrix",V.inverse.transpose);

So my approach is kinda, full of conversions which I don’t think they are very optimal.
Performance wise, I can’t tell because it is rendering at 2-3ms so it is not too bad on my PC but I got i7 930 running with GTX 660.

what did you do with the following line of code

//float4 vspPosReflectT = mul (UNITY_MATRIX_P,viewPos+float4(vspReflect,1));

float4 vspPosReflectT = mul (_ProjMatrix, viewPos +float4(vspReflect,1));

float3 vspPosReflect = vspPosReflectT.xyz / vspPosReflectT.w;

are you dividing with .w or .z also is Linear01Depth a unity function? This has been buggin me:)

Here is web player of what I am working on… Use Maya like navigation. (use alt + mouse click drag)

http://www.castorlee.com/script/shader/SSR/WebPlayer.html

Superjay, I am dividing with .w Nothing has changed there…Also Linear01Depth is function from Unity included in UnityCG.cginc file.

As you can see from the player, that I have issue with false reflection catch… If I make my depth compare bit more tighter, then I get very noisy result around that false reflection. The web player version is the one with relaxed depth compare. I need to come up with better ray marching technique to get rid of false reflection without too much noise. No blur filter will get rid of that…

It looks nice. What’s causing the cylindrical shaped reflection error with the spheres?
I can’t believe this is running on my 8800GTS with WinXP :sunglasses:

You have a PM.

It is caused by the what I call “false reflection”. Basically, when you want to get a reflection pixel on certain area, the reflection is supposed to point to the “back” of the model you can see. Which you can’t get because of the obvious limitation of the screen space reflection. In SSR you can only reflect what you can see in the screen, nothing behind the back of model or off the screen etc…

I tell my ray march to stop when the difference between reflection ray’s depth and depth from depth map is less than certain value. When I let the difference to be tight, I get very noisy artifacts and if I let it be bit more forgiving, then it shows that cylindrical error. (it thinks it found right reflection point but in fact it is not ← but shader doesn’t know it is wrong…)

If I can be more super accurate with my depth compare, or come up with some other way of doing ray marching ( or no ray march at all ) then this issue may be fixed…

OR… I am thinking rendering the scene again from the opposite of the camera angle and then use that as pixel information at the back. But this is rendering the scene twice so it could be a crazy idea. hehe.

Once again nice job! Is the glow around the meshes in the WebPlayer caused by SSR? SSAO? It seems really noticeable but the reflections are still nice! Maybe its just a sampling issue.

The glow is probably caused by doing blur pass on the reflection.

Edit:

I tried to magnify the z value before I do the compare so the difference gets proportionally larger as it goes…
It has helped to get rid of “some” false reflection but it still makes the pixels at around those border of max z difference to become very noisy.

When I want to do very tight z compare, some “good” reflection gets filtered out… I think this contributes to the noisy reflection.

Still thinking about the ways to improve ray march and z compare…

Also , I was thinking, maybe non pixel perfect fetch of depth map is causing this?
I am sure at the moment, depth is sampled at texture filtered method so maybe I am getting some average depth…
I think it could be the same issue Brn had…

hummmmm…