I am trying to simulate a LIDAR in Unity, i.e. I am trying to create a pointcloud based on the surfaces around the sensor. In other words, the distance of a large set of rays shot from some point.
I figured I can approach this with a regular camera. Cameras also compute view depth for each pixel, which is already very close to what I’m looking for.
Question: how can I extract the view depth information from a camera render and process it with a script?
Since most rendering things live on the GPU, I am almost certain I will need custom shaders to get anything done.
What I got so far: I have a scene with the main camera and a LIDAR camera. The LIDAR camera renders to a RenderTexture, which is used by a LIDARMaterial. The LIDARMaterial also has a LIDARShader for processing.
Then, at runtime I render the LIDARCamera and use Graphics.Blit to process the LIDARCamera result with the LIDARMaterial (effectively applying my custom shader) and store the result in a new texture.
For debugging purposes I put this texture on a test material which I apply to a ‘TV Screen’ inside the scene:
Now I figured I would modify the shader to simply render the view depth as e.g. grayscale, like:
Where I’m stuck: I cannot get any depth information from the shader. What I have now:
Shader "Unlit/LIDARShader"
{
Properties
{
_MainTex ("Texture", 2D) = "white" {}
}
SubShader
{
Tags { "RenderType"="Opaque" }
Pass
{
CGPROGRAM
#pragma vertex vert
#pragma fragment frag
#include "UnityCG.cginc"
struct appdata
{
float4 vertex : POSITION;
float2 uv : TEXCOORD0;
};
struct v2f
{
float2 uv : TEXCOORD0;
float depth : TEXCOORD1;
float4 vertex : SV_POSITION;
};
sampler2D _MainTex;
float4 _MainTex_ST;
uniform sampler2D _CameraDepthTexture;
v2f vert (appdata v)
{
v2f o;
o.vertex = UnityObjectToClipPos(v.vertex);
o.depth = o.vertex.z;
o.uv = TRANSFORM_TEX(v.uv, _MainTex);
return o;
}
fixed4 frag (v2f i) : SV_Target
{
fixed4 col = tex2D(_MainTex, i.uv);
// col.r = 1.0;
col.r = saturate(tex2D(_CameraDepthTexture, i.uv));
return col;
}
ENDCG
}
}
Fallback "ShadowCaster"
}
But the result from the depth texture samples is always 0. What’s going wrong here?
I am using Unity 2022.3.5f1 with the default rendering pipeline.
Full project is attached too.
Other reading:
There is the manual about Cameras and Depth: Unity - Manual: Cameras and depth textures
Pretty much the same question but from 2012: How to access rendered depth buffer properly?
9413600–1318526–LidarProject.unitypackage (15.6 KB)
