Camera distortion Post-Processing & ScreenPointToRay

Unity Engine
, ,
1

When using camera distortion volume effects calling ScreenPointToRay returns values without taking into acconut the distorted viewport…

is there a way to solve this? for example a way to apply the visual distortion to the coordinates too?
For example the Lens Distorion volume, hen set to high disortion values, it makes ScreenPointToRay useless… it would be cool if one could get the Input.mousePosition and apply some kind of formula to it to mimic the pixel distortion roduced by the volume effect.

Anyone skilled in math knows how to apply the LensEffect to a screen mouse coordinate?

2

According to chatGTP something like this could be the answer… i haven’t tested this since i should have to create a custon lens distortion effect to expose th K123 coeficients…

using UnityEngine;

public class LensDistortionRayCasting : MonoBehaviour
{
    public Camera mainCamera;
    public float k1;
    public float k2;
    public float k3;

    void Update()
    {
        // Get mouse position in screen coordinates
        Vector3 mousePosition = Input.mousePosition;

        // Convert to normalized device coordinates (NDC)
        Vector2 ndc = new Vector2(mousePosition.x / Screen.width * 2f - 1f, mousePosition.y / Screen.height * 2f - 1f);

        // Apply lens distortion effect
        Vector2 distortedNDC = ApplyLensDistortion(ndc, k1, k2, k3);

        // Convert back to screen coordinates
        Vector3 distortedScreenPosition = new Vector3((distortedNDC.x + 1f) * 0.5f * Screen.width, (distortedNDC.y + 1f) * 0.5f * Screen.height, mousePosition.z);

        // Cast ray from the post-distorted screen position
        Ray ray = mainCamera.ScreenPointToRay(distortedScreenPosition);

        // Perform raycast operations as needed
        // ...
    }

    Vector2 ApplyLensDistortion(Vector2 ndc, float k1, float k2, float k3)
    {
        float r2 = ndc.x * ndc.x + ndc.y * ndc.y;
        float distortionFactor = 1.0f + k1 * r2 + k2 * r2 * r2 + k3 * r2 * r2 * r2;
        return ndc * distortionFactor;
    }
}
3

is this true? apparently there’s no easy way to invert the distorion effect. ChatGPT:

To invert this distortion, you would need to find a set of equations that, given a distorted point, can solve for the undistorted point. Unfortunately, due to the complexity of the distortion equation involving higher-order terms, there’s no simple closed-form solution to invert it directly.

Lens distortion is commonly corrected using iterative methods or optimization algorithms to find the undistorted point that would have caused the distorted point to appear where it is. Some popular algorithms for lens distortion correction include Newton-Raphson, Levenberg-Marquardt, or other non-linear optimization techniques. These methods typically involve iteratively refining the solution until the undistorted point is found.

4

i can’t believe there’s no way to do a simple raycast from a distorted screen.

If you use a fullscreen shader to create for example a fisheye effect, you lose the posibility to use screen raycasts to interact with the scene. because the raycasts doesn’t follow the distortion and will point to the scene in the direction as if the distortion was not there…

5

Render a uv coordinate texture with your distorted settings and save it as its rendered. Save that texture and use it as a input on your camera to revert the transformation, and raycast from the correct position. Whenever coordinate your click fetches on the screen from the texture is the original screen position of that pixel

I know this is a 5 months old question, but who knows, maybe this hacky solution could help you still