Hey everyone, I’m having a bit of a ‘not quite understanding how this works’ situation here.
I’m suppose to do a simple app that you first take a picture of your wall with a camera (or add from your library), after that you would mark 4 points that correspond the walls dimensions (for sake of simplicity let’s say you mark all four corners of the wall). With these four points I would calculate the camera pose in relative to the wall (I will also ask the user to input the width and height of the marked area, in this example the whole wall). At least currently I’m forcing the left and right edge of the selection to be completely vertical.
After this is done, I would then take the given information, put them in OpenCV (currently I use the free OpenCV plus Unity) SolvePnP()-function and use the output to rotate and translate a plane mesh or the camera in the world, so that the camera pose is correct.
So currently I do all this, looking something like this:
First I take the world positions of a plane and screen positions of the points the user has given, where op is objectPoints, ip is imagePoints.
MatOfPoint3f op = new MatOfPoint3f();
for (int i = 0; i < m_meshObject.mesh.vertexCount; i++) {
op.Add(
new Point3f(
m_meshObject.mesh.vertices[i].x,
m_meshObject.mesh.vertices[i].y,
m_meshObject.mesh.vertices[i].z
)
);
}
MatOfPoint2f ip = new MatOfPoint2f();
for (int i = 0; i < guideLinePoints.Count; i++) {
Vector2 screenPos = m_camera.WorldToScreenPoint(guideLinePoints[i]);
ip.Add(
new Point2f(
screenPos.y,
screenPos.x
)
);
}
I do also a camera matrix based on some examples I’ve found online and I ignore distortion coefficients. These two don’t really matter, since the result doesn’t have to be perfect, like in an AR app.
So with these I run SolvePnP(). After this, I do a few conversions.
Mat Rvec = new Mat();
Mat Tvec = new Mat();
Mat rvec = new Mat(1, 3, MatType.CV_64FC1);
Mat tvec = new Mat(1, 3, MatType.CV_64FC1);
Cv2.SolvePnP(
op,
ip,
camMat,
distCoeffs,
rvec,
tvec
);
rvec.ConvertTo(Rvec, MatType.CV_32F);
tvec.ConvertTo(Tvec, MatType.CV_32F);
Mat rotMat = new Mat(3, 3, MatType.CV_64FC1);
Cv2.Rodrigues(Rvec, rotMat);
After this I don’t really know anymore what to do. With these I should be able to rotate and translate my camera, but I don’t know how. I’ve tried to look at all sorts of examples and tutorials (most of which are for OpenCV in Python or C++ and these don’t help with the Unity transform-stuff).
So with what I have now, screen positions like these:
I get values like these:
I have no idea if these even look like values I should be getting, but I just added these if they are of any help.
TL: DR; I’m trying to use OpenCV SolvePnP() to get camera pose from a picture of a wall and manually added 4 points indicating the corners of the wall in the picture. I get values from SolvePnP(), but I’m at loss on how to convert these matrices to Unity transform.position and transform.rotation.
If anyone could help me it would be super great, I’m losing my mind here…