Trying to get VR passthrough work with sentis

I have an ONNX model that takes an image as input, transforms it using a computational model, and outputs a transformed image. I can explain more what the model does if anyone wants to know more!
But TLDR I want to get the camera texture from the VR pass through cameras to be used as an input and then display the output of the model on the headset.
I was using Barracuda until now and it was working pretty well but soon I hit a wall with more complex ONNX models. now I can’t even run my old onnx models with sentis.
Anyone working on this kind of project? would love to get some insights, specifically how to convert an image output tensor to a texture

Which VR is it. In Oculus 2 for example, you don’t have access to the passthrough image. I think this is for privacy reasons so apps don’t spy on your house. For a regular webcam you can try this face detection model on Hugging Face as an example of using the webcam

using a vive pro eye, thanks for the example I will try it and get back if it works!

What do you mean by this?
check this sample out

I think the problem I am facing is the model works on a static rawimage but for vr headset I can’t use webcamtexture as it would stop tracking instead I have to use steamVR_trackedcamera texture. It seems like something goes wrong there. I will update if I figure it out

Just a guess, but could it be that the image is in side-by-side mode or that it is warped in order to be viewed in the lenses? That would affect the detection a bit. Also if the image is very noisy you can try under brighter lights.

I’ll make a note that we should probably have some tutorials about how to get this working in XR/VR in the future.

3 Likes