Problem with custom depth map

Questions & Answers
1

Hello everyone,

I’m working on a simple simulation for activities of daily living. One part of this simulation is the generation of depth maps which we use for deep learning evaluations a.s.o… A problem is that we use the fisheye projection approach from Bourke et al. (http://paulbourke.net/dome/unity3d/) and a limit is that we can’t use the depth maps created for each camera. Long story short, I’ve written a simple shader that seems to work except the case if some character is laying in the bed. For an unknown reason, the person in the bed is closer to the ground than the surface of the bed in the depth map. See screenshot below:

153621-depth.jpg
153621-depth.jpg1000×1000 66.1 KB 

# Shader Code

float4x4 _fisheyeView;
inline float CustomDepth(float3 vertex) {
    float4x4 mat = mul(_fisheyeView, UNITY_MATRIX_M);
    float3 v = mul(mat, vertex);

    return 1 - (v.z / _fisheyeView._m13);
}

Has someone an explanation for this?

2

Try the following;

inline float CustomDepth (float3 vertex) 
{
    float3 v = mul (_fisheyeView, float4 (mul (unity_ObjectToWorld, float4 (vertex, 1)).xyz, 1));
 
    return 1 - (v.z / _fisheyeView._m13);
}
  1. UNITY_MATRIX_M can sometimes be undefined, whereas unity_ObjectToWorld should always work.
  2. You were multiplying a float3 by a float4x4, meaning the matrix would have been truncated to a float3x3 and the vertex would have been interpreted as a vector rather than a position.
  3. It should be faster to multiply a point by a matrix than it is to multiply two matrices.

Not certain it will fix the problem, but if i had to guess why it was happening, points 1 and 2 would be my first thoughts.