How to reproject rendertexture from CameraA to match what CameraB sees.

Graphics Shaders & Materials
,
1

Let me give the detailed infomation:
I have a pair of AR glasses and am trying to align the RGBcamera image with what i really see.

First for now I want to simulate the scenary in UnityEditor, so I created two cameras A and B in the scene.
Camera A render the scene to a rendertexture.
With the rendertexture, I placed a Quad to fulfil Camera A’s nearplane.
Camera B’s position and rotation are different from Camera A’s.

My idea is:
9244191--1292121--upload_2023-8-23_21-0-0.png
Here is the code and shader, but the result seems not right.
Can you guys give me some ideas?

9244191--1292112--upload_2023-8-23_20-55-54.png
9244191--1292112--upload_2023-8-23_20-55-54.png933×106 9.23 KB

9244191--1292115--upload_2023-8-23_20-56-50.png
9244191--1292118--upload_2023-8-23_20-58-41.png

2

I probably found why the result is not right. I am using the on-axis perspective projection matrix.
But the problem i am dealing with is off-axis perspective projection.
Am I right?
Thanks if anyone could give me suggestions.

3

The short version is: you can’t do that.

The long version is: you have to project onto a 3D surface. Think of using a real life video projector on the side of a building or a statue.

4

Thanks for your explaination!
Below is my conclusion:
The Depth infomation is Lost in RGB Camera Image, so It is not impossible to precisely remap pixels between two views.

5

Correct. This is why Meta, Apple, and others that have external cameras also have methods to mapping real world geometry around it, and project the RGB camera’s view onto that geometry when displayed to the user.