Help Reproducing Screen To World Conversion

Unity Engine
1

Hi, I’m trying to recreate the ScreenPointToRay function in code using the camera MVP matrices.

What am I doing wrong below that is leading to different results compared with the built-in camera function?

      var model       = camera.transform.localToWorldMatrix;
      var view        = camera.worldToCameraMatrix;
      var projection  = camera.projectionMatrix;

      Matrix4x4 MVP = projection*view*model;

      float4 mousePos_clipSpace = new float4(
                                    ((mouseX * 2.0f) / Screen.width) - 1.0f,
                                    (1.0f - (2.0f * mouseY) / Screen.height),
                                    0.0f,
                                    1.0f);

      float4 wspace = MVP.inverse * mousePos_clipSpace;
      wspace /= wspace.w;
     
      // Why don't these match?

      float3 ray_Wrong = math.normalize (new float3 (wspace.x, wspace.y, wspace.z));

      Ray startCameraRay = camera.ScreenPointToRay (new Vector3 (mouseX, mouseY, 0.0f));
      float3 ray_Correct = startCameraRay.direction;
2

Don’t use the “model” matrix, it’s meaningless here as the camera doesn’t really have one. Using the transform.localToWorldMatrix for anything rendering related is actually wrong as it does not necessarily match the model space used for rendering, but it’s doubly wrong here. You’re trying to get a ray in world space, and the camera’s cameraToWorldMatrix (or the inverse of the worldToCameraMatrix like you’re using) is all that’s needed for getting a camera space ray into world space.

1 Like
3

Thanks for replying, @bgolus . In the end I got it working with the following code should it interest anyone. If there’s a better/quicker way let me know.

fyi this was handy so that I could call it from a bursted job using the new DOTs packages.

public static float3 ScreenToWorldRay (float2 screenPosition,
                                         float screenWidth,
                                         float screenHeight,
                                         float4x4 cameraToWorld,
                                         float4x4 cameraProjectionInverse,
                                         float3 cameraOrigin)
  {
    screenPosition = new float2 (screenPosition.x, screenHeight - screenPosition.y);

    Vector4 clipSpace = new Vector4 (((screenPosition.x * 2.0f) / screenWidth) - 1.0f, (1.0f - (2.0f * screenPosition.y) / screenHeight), 0.0f, 1.0f);

    float4 viewSpace = math.mul (cameraProjectionInverse, clipSpace);
    viewSpace /= viewSpace.w;
   
    float4 worldSpace = math.mul (cameraToWorld, viewSpace);

    float3 worldDirection = math.normalize (worldSpace.xyz - cameraOrigin);
   
    return worldDirection;
  }
1 Like