Rendering virtual objects within a real scene

Questions & Answers
1

I have depth maps and color images (both of them are from a camera), and I create some virtual objects to interact with real objects in the scene. How to render both of them (in Unity)? Maybe a shader will work. But I am relatively new to Cg. In a pixel shader, how to compare depth values of different objects which are projected to one same pixel? Thanks.

So what in my mind is this: A color image with depth data at each pixel (get from a camera), and several virtual objects in Unity. I want to overlay them. I want a pixel shader to check depth buffer for every pixel. If the depth value of the pixel from color image is smaller, rendering that pixel. Or if the virtual object is closer, rendering the virtual object.

2

I haven't seen any code from you yet, not going to write a shader for you, but your idea seems reasonable. Would be easy as a post-processing effect, just plug in all four images and select based on the depths. Doing this will require Unity Pro. You can use the Depth of Field image effect as an example of how to obtain the camera depth texture from script and access the depth buffer from an image effect shader. A depth texture appears to the shader as a texture of single unorm value in the red channel. Simple comparing the depths to see which is smaller should be simple enough if that's all you need.

http://docs.unity3d.com/Manual/SL-CameraDepthTexture.html

http://docs.unity3d.com/Manual/script-DepthOfField.html