Hello dear Hololens 2 fellows, first of all, thank you very much for reading through my issue!
What I am trying right now is the following:
- Obtaining a photo from the locatable camera of the Hololens 2.
- Finding a specific point on the photo texture (e.g. a corner of a QR code).
- Get the location of this point within 3D space of the scene.
My base code is derived from this example code: Photo Video camera in Unity - Mixed Reality | Microsoft Learn
I can easily perform steps 1) and 2). I also could verify that both step 1) and 2) are working currently, so we can assume that.
Step 3 is the hard part now at which I am stuck. Currently, I am trying to cast a ray through the texture point from the camera (which however seems to be the main camera != the locatable camera locationwise) towards the world mesh. I also tried different things (I am getting also the cameraToWorldMatrix from the locatable camera), however, after several days without a solution, I would appreciate it if someone could point me in the right direction.
My question is: How can I get the 3D location within the scene corresponding to one particular pixel of the photo texture that I get from the PhotoCapture functionality of the locatable camera of the Hololens 2?
Thanks so much!