Cubemap reflections in tangent space

[ilya_ca]

Hi,
I have some misunderstandings about tangent space. I have a CG shader that performs cubemap reflections. It works fine when using world-space reflections on a horizontal plane. I want to make it work on an arbitrary mesh, so I’ll have to transform the view direction into tangent space. Here’s a snippet from my shader:

v2f vert (appdata_tan v)
{
    v2f o;
    o.pos = mul (UNITY_MATRIX_MVP, v.vertex);
    
    TANGENT_SPACE_ROTATION;
    o.viewDir = mul (rotation, ObjSpaceViewDir(v.vertex)); 
}

fixed4 frag (v2f i) : COLOR
{
	half3 refl = reflect( normalize(i.viewDir), half3(0.0, 1.0, 0.0) );
	refl.x = -refl.x;	
	texcol.rgb = texCUBE( _Cube , refl ).rgb;

	return texcol;
}

What am I doing wrong and how can I make the reflections work in tangent space?

Thanks!

[ilya_ca]

I’ve just checked the result reflection and it seems that the whole reflection is rotated 90 degrees in the ZY plane. On the left of my reflective surface I can see the bottom of the cubemap instead of the left side of the cubemap. For some reason the tangent space viewDir is off by 90 degrees.

[ilya_ca]

If I rotate the object by 90 degrees, reflections start being correct.

[Martin-Kraus]

The difference that you see is only in relation to the RenderFX/Skybox shader (which is unfortunately the shader that is used by the skyboxes in the standard assets). Just use the RenderFX/Skybox Cubed shader and you should be fine.

Note to the Unity team: as far as I can tell RenderFX/Skybox and RenderFX/Skybox Cubed are not perfectly compatible (at least not with the way that cube textures are suggested to be built). You should at least warn users.

[ilya_ca]

Actually I’m using my own cubemap which has nothing to do with the skybox…

samplerCUBE _Cube;

It used to work correctly until I introduced the tangent-space rotation:

TANGENT_SPACE_ROTATION;
o.viewDir = mul (rotation, ObjSpaceViewDir(v.vertex));

I suppose the problem has something to do with the way I’m doing the rotations. It seems to me that the view vector is incorrectly rotated for some reason.

I tried to output the view direction vector in my fragment program in the following way:

half3 normViewDir = normalize(i.viewDir);
return fixed4(normViewDir.x, normViewDir.y, normViewDir.z, 1.0) * 0.5 + 0.5 * fixed4(1.0, 1.0, 1.0, 1.0);

And it outputs a totally different color at the same viewing angle (for tangent space and for world-space view direction vector). So I suppose that viewDir results in a completely different vector. But why?

[Martin-Kraus]

Oh, OK, I have no idea about this tangent space rotation. Sorry. (But now I do wonder: why do you want to use whatever “tangent space rotation” is?)

[Martin-Kraus]

OK, out of curiosity, I’ve looked at the definitions in UnityCG.cginc:

// Computes object space view direction
inline float3 ObjSpaceViewDir( in float4 v )
{
	float3 objSpaceCameraPos = mul(_World2Object, float4(_WorldSpaceCameraPos.xyz, 1)).xyz * unity_Scale.w;
	return objSpaceCameraPos - v.xyz;
}

// Declares 3x3 matrix 'rotation', filled with tangent space basis
#define TANGENT_SPACE_ROTATION \
	float3 binormal = cross( v.normal, v.tangent.xyz ) * v.tangent.w; \
	float3x3 rotation = float3x3( v.tangent.xyz, binormal, v.normal )

Thus, in my understanding ObjSpaceViewDir returns the view direction in object space.
And the matrix “rotation” transforms from a local tangent space (with the (0,0,1) direction along the normal) to object space.

Thus, transforming the result of ObjSpaceViewDir with the matrix “rotation” doesn’t make any sense to me.

But maybe you have a reason for it, or I’m missing something.

EDIT: I was missing that Cg uses a row-major matrix format (I’m used to the column-major format of GLSL); thus, rotation specifies the transposed matrix of what I thought it would be and thus the matrix specifies the inverse matrix (or at least a scaled version of it). Thus, “rotation” transforms from the object space to the tangent space and therefore the result has to be treated like a vector in tangent space.

[ilya_ca]

Actually I want to use a normalmap in my shader, and the normalmap is always in tangent space. So I’ll have to transform the view direction vector into tangent space to make it work properly. Here’s a good explanation about normal maps in tangent and object space: http://wiki.polycount.com/NormalMap/

[Farfarer]

You may find that you have to convert the normal map back from tangent to world space (and pass through a world space view direction) in order to get this working.

Or possibly rotate your reflection vector from object to world space?

half3 refl = reflect( normalize(i.viewDir), half3(0.0, 1.0, 0.0) );
refl.x = -refl.x;
refl = mul((float3x3)_Object2World, refl).xyz;
texcol.rgb = texCUBE( _Cube , refl ).rgb;

Tangent space is a rotation from object space, so if your model is imported from 3DS Max, for example (with Z-up) it’s object space will be rotated 90 degrees from world space, throwing everything off.

You should be able to take the reflection vector back to world space using the built-in _Object2World matrix… - do the tangent space vectors cancel each other out? I dunno (I’m tired and vector maths is beyond me at the moment).

[ilya_ca]

@Martin Kraus

Thinking about it a 2nd second, I think I know what is going wrong:

fixed4 frag (v2f i) : COLOR
{
    half3 refl = reflect( normalize(i.viewDir), half3(0.0, 1.0, 0.0) );
    refl.x = -refl.x;   
    texcol.rgb = texCUBE( _Cube , refl ).rgb;
    return texcol;
}

i.viewDir is in tangent space, I think that (0,0,1) is the normal direction in tangent space, thus, I guess you should write: half3 refl = reflect( normalize(i.viewDir), half3(0.0, 1.0, 0.0) );
And you might have to use -i.viewDir (I never know in which direction this vector is defined, “reflect” expects the vector in the direction in which you are looking; however, the view direction is often defined as the direction to the camera.

“refl.x = -refl.x” doesn’t make any sense to me.

In order to use refl for the cube map, you have to transform it to world space, i.e. multiply it with the transpose of rotation (probably this way: mul(refl, rotation) ) to go to object space, and then with _Object2World to get to world space: mul(_Object2World, mul(refl, rotation))

Hope that works

This makes a lot of sense! The “texCUBE(…)” function expects the vector to be in world space, however I’m supplying it with a vector in tangent space (which is of course incorrect).

So I think my best bet would be:

• For lighting calculations that don’t require cubemap reflections I could translate the viewDir into tangent space (once in a vertex shader) in vertex shader.
• For shaders that require cubemap reflections it would be best to translate the per-pixel normal (retrieved from the normal map) into world space in the fragment shader.

Because otherwise if I do cubemap reflections then I would have to translate the viewDir into tangent space in the vertex shader. And then I would have to translate the reflected vector back into world space in the pixel shader.

[Farfarer]

I have the code for that to hand;

struct v2f
{
	float4	pos : SV_POSITION;
	float2	uv : TEXCOORD0;
	float3	normal : TEXCOORD1;
	float3	tangent : TEXCOORD2;
	float3	binormal : TEXCOORD3;
	float3	viewDir : TEXCOORD4;
}; 

v2f vert (appdata_tan v)
{
	v2f o;
	o.pos = mul(UNITY_MATRIX_MVP, v.vertex);
	o.uv = v.texcoord.xy;
	o.normal = mul(_Object2World, float4(v.normal, 0)).xyz;
	o.tangent = v.tangent;
	o.binormal = cross(o.normal, o.tangent) * v.tangent.w;
	o.viewDir = WorldSpaceViewDir(v.vertex);
	return o;
}

sampler2D _NormalTex;
samplerCUBE _CubeTex;

float4 frag(v2f i) : COLOR
{
	float3 normal = UnpackNormal(tex2D(_NormalTex, i.uv));
	float3 normalW = (i.tangent * normal.x) + (i.binormal * normal.y) + (i.normal * normal.z);
	reflection = texCUBE(_CubeTex, reflect(normalize(-i.viewDir), normalW));
}

[ilya_ca]

@Farfarer

I have the code for that to hand;

Thanks a lot, that’s exactly what I was looking for!