Editor raycast against scene meshes without collider; Editor Select-object using GUI coordinate

Here is a piece of code allowing you to do what the title says.
Bwa!!

Source:
https://github.com/MattRix/UnityDecompiled/blob/master/UnityEditor/UnityEditor/HandleUtility.cs

I’ve improved it to include object picking as well

//powerful class, allows to detect intersection with mesh, without requiring any collider, etc
//Works in editor only
//
// Main Author https://gist.github.com/MattRix
// Igor Aherne improved it to include object picking as well   facebook.com/igor.aherne
//https://github.com/MattRix/UnityDecompiled/blob/master/UnityEditor/UnityEditor/HandleUtility.cs
[InitializeOnLoad]
public static class EditorHandles_UnityInternal {
    static Type type_HandleUtility;
    static MethodInfo meth_IntersectRayMesh;
    static MethodInfo meth_PickObjectMeth;

    static EditorHandles_UnityInternal() {
        var editorTypes = typeof(Editor).Assembly.GetTypes();

        type_HandleUtility = editorTypes.FirstOrDefault(t => t.Name == "HandleUtility");
        meth_IntersectRayMesh = type_HandleUtility.GetMethod("IntersectRayMesh",
                                                              BindingFlags.Static | BindingFlags.NonPublic);
        meth_PickObjectMeth = type_HandleUtility.GetMethod("PickGameObject",
                                                            BindingFlags.Static | BindingFlags.Public,
                                                            null,
                                                            new [] {typeof(Vector2), typeof(bool)},
                                                            null);
    }


    //get a point from interected with any meshes in scene, based on mouse position.
    //WE DON'T NOT NEED to have to have colliders ;)
    //usually used in conjunction with  PickGameObject()
    public static bool IntersectRayMesh(Ray ray, MeshFilter meshFilter, out RaycastHit hit) {
        return IntersectRayMesh(ray, meshFilter.mesh, meshFilter.transform.localToWorldMatrix, out hit);
    }

    //get a point from interected with any meshes in scene, based on mouse position.
    //WE DON'T NOT NEED to have to have colliders ;)
    //usually used in conjunction with  PickGameObject()
    public static bool IntersectRayMesh(Ray ray, Mesh mesh, Matrix4x4 matrix, out RaycastHit hit) {
        var parameters = new object[] { ray, mesh, matrix, null };
        bool result = (bool)meth_IntersectRayMesh.Invoke(null, parameters);
        hit = (RaycastHit)parameters[3];
        return result;
    }

 
 
//select a gameObject in scene, based on mouse position.
    //Object DOES NOT NEED to have to have colliders ;)
    //If you DON'T want object to be included into  Selection.activeGameObject,
    //(parameter works only in gui functions and scene view delegates) specify updateSelection = false
    public static GameObject PickGameObject(Vector2 position, bool updateSelection = true, bool selectPrefabRoot = false) {

        if (updateSelection == false && Event.current != null) {
            int blocking_ix = GUIUtility.GetControlID(FocusType.Passive);
            HandleUtility.AddDefaultControl(blocking_ix);
            GUIUtility.hotControl = blocking_ix; //tell unity that your control is active now, so it won't do selections etc.
        }

        GameObject pickedGameObject = (GameObject)meth_PickObjectMeth.Invoke(null,
                                                       new object[] { position, selectPrefabRoot });

        return pickedGameObject;
    }
 
}
12 Likes

It would be great if Unity could expose a more optimized version of this method. I’m not sure what all it’s doing internally, but it is quite slow for larger meshes. It appears that PolyBrush uses their own custom ray-triangle intersection methods to avoid having to use reflection to get to this.

Did IntersectRayMesh stop working in unity 2020? It doesn’t hit anything anymore for me. Was working fine in 2019.

It seem the issue is the code to convert the mouse position to a ray is not working properly anymore. Anybody got a working version? Right now the ray is quite offset from the mouse.
EDIT
Now it restarted working. I think some editor asset is screwing up something.

Use with caution though. I’ve got this crashing the UnityEditor (2019 LTS) in a somewhat random manner (actually it crashes rarely but it does). Calling it from SceneView.duringSceneGui:

0x00007FF629795EE7 (Unity) HandleUtilityBindings::IntersectRayMesh
0x00007FF62977CB5E (Unity) HandleUtility_CUSTOM_IntersectRayMesh_Injected
0x0000017376549C8B (Mono JIT Code) (wrapper managed-to-native) UnityEditor.HandleUtility:IntersectRayMesh_Injected (UnityEngine.Ray&,UnityEngine.Mesh,UnityEngine.Matrix4x4&,UnityEngine.RaycastHit&)
0x000001737654941B (Mono JIT Code) UnityEditor.HandleUtility:IntersectRayMesh (UnityEngine.Ray,UnityEngine.Mesh,UnityEngine.Matrix4x4,UnityEngine.RaycastHit&)
0x0000017376549ACD (Mono JIT Code) (wrapper runtime-invoke) <Module>:runtime_invoke_bool_Ray_object_Matrix4x4_intptr& (object,intptr,intptr,intptr)
0x00007FFDD3CED6B0 (mono-2.0-bdwgc) [c:\build\output\unity-technologies\mono\mono\mini\mini-runtime.c:2809] mono_jit_runtime_invoke
0x00007FFDD3C72912 (mono-2.0-bdwgc) [c:\build\output\unity-technologies\mono\mono\metadata\object.c:2921] do_runtime_invoke
0x00007FFDD3C7BB42 (mono-2.0-bdwgc) [c:\build\output\unity-technologies\mono\mono\metadata\object.c:3073] mono_runtime_invoke_checked
0x00007FFDD3C7C2D9 (mono-2.0-bdwgc) [c:\build\output\unity-technologies\mono\mono\metadata\object.c:5264] mono_runtime_try_invoke_array
0x00007FFDD3C7BAD6 (mono-2.0-bdwgc) [c:\build\output\unity-technologies\mono\mono\metadata\object.c:5142] mono_runtime_invoke_array_checked
0x00007FFDD3C20384 (mono-2.0-bdwgc) [c:\build\output\unity-technologies\mono\mono\metadata\icall.c:3358] ves_icall_InternalInvoke
0x00000173272EBA86 (Mono JIT Code) (wrapper managed-to-native) System.Reflection.MonoMethod:InternalInvoke (System.Reflection.MonoMethod,object,object[],System.Exception&)
0x00000173272EA43B (Mono JIT Code) System.Reflection.MonoMethod:Invoke (object,System.Reflection.BindingFlags,System.Reflection.Binder,object[],System.Globalization.CultureInfo)
0x00000173272EA0FF (Mono JIT Code) System.Reflection.MethodBase:Invoke (object,object[])
...
0x000001737565D76E (Mono JIT Code) UnityEditor.SceneView:CallOnSceneGUI ()
0x0000017375659063 (Mono JIT Code) UnityEditor.SceneView:HandleSelectionAndOnSceneGUI ()
0x0000017375612EEB (Mono JIT Code) UnityEditor.SceneView:OnGUI ()
0x0000017326E613B8 (Mono JIT Code) (wrapper runtime-invoke) object:runtime_invoke_void__this__ (object,intptr,intptr,intptr)
0x00007FFDD3CED6B0 (mono-2.0-bdwgc) [c:\build\output\unity-technologies\mono\mono\mini\mini-runtime.c:2809] mono_jit_runtime_invoke
0x00007FFDD3C72912 (mono-2.0-bdwgc) [c:\build\output\unity-technologies\mono\mono\metadata\object.c:2921] do_runtime_invoke
0x00007FFDD3C7BB42 (mono-2.0-bdwgc) [c:\build\output\unity-technologies\mono\mono\metadata\object.c:3073] mono_runtime_invoke_checked
0x00007FFDD3C7C2D9 (mono-2.0-bdwgc) [c:\build\output\unity-technologies\mono\mono\metadata\object.c:5264] mono_runtime_try_invoke_array
0x00007FFDD3C7BAD6 (mono-2.0-bdwgc) [c:\build\output\unity-technologies\mono\mono\metadata\object.c:5142] mono_runtime_invoke_array_checked
0x00007FFDD3C20384 (mono-2.0-bdwgc) [c:\build\output\unity-technologies\mono\mono\metadata\icall.c:3358] ves_icall_InternalInvoke
0x00000173272EBA86 (Mono JIT Code) (wrapper managed-to-native) System.Reflection.MonoMethod:InternalInvoke (System.Reflection.MonoMethod,object,object[],System.Exception&)
0x00000173272EA43B (Mono JIT Code) System.Reflection.MonoMethod:Invoke (object,System.Reflection.BindingFlags,System.Reflection.Binder,object[],System.Globalization.CultureInfo)
0x00000173272EA0FF (Mono JIT Code) System.Reflection.MethodBase:Invoke (object,object[])
0x0000017329735CAB (Mono JIT Code) UnityEditor.HostView:Invoke (string,object)
0x0000017329735B1B (Mono JIT Code) UnityEditor.HostView:Invoke (string)
0x0000017351E8E2AB (Mono JIT Code) UnityEditor.HostView:InvokeOnGUI (UnityEngine.Rect,UnityEngine.Rect)
0x0000017351E8DD63 (Mono JIT Code) UnityEditor.DockArea:smile:rawView (UnityEngine.Rect,UnityEngine.Rect)
0x0000017351E7A55B (Mono JIT Code) UnityEditor.DockArea:OldOnGUI ()
0x0000017351E42B52 (Mono JIT Code) UnityEngine.UIElements.IMGUIContainer:smile:oOnGUI (UnityEngine.Event,UnityEngine.Matrix4x4,UnityEngine.Rect,bool,UnityEngine.Rect,System.Action,bool)
0x0000017351E405C3 (Mono JIT Code) UnityEngine.UIElements.IMGUIContainer:HandleIMGUIEvent (UnityEngine.Event,UnityEngine.Matrix4x4,UnityEngine.Rect,System.Action,bool)
0x00000173755CD093 (Mono JIT Code) UnityEngine.UIElements.IMGUIContainer:smile:oIMGUIRepaint ()
0x00000173755C8EA6 (Mono JIT Code) UnityEngine.UIElements.UIR.RenderChainCommand:ExecuteNonDrawMesh (UnityEngine.UIElements.UIR.DrawParams,bool,single,System.Exception&)
0x00000173755C2853 (Mono JIT Code) UnityEngine.UIElements.UIR.UIRenderDevice:EvaluateChain (UnityEngine.UIElements.UIR.RenderChainCommand,UnityEngine.Rect,UnityEngine.Matrix4x4,UnityEngine.UIElements.PanelClearFlags,UnityEngine.Texture,UnityEngine.Texture,UnityEngine.Texture,single,Unity.Collections.NativeArray`1<UnityEngine.UIElements.UIR.Transform3x4>,Unity.Collections.NativeArray`1<UnityEngine.Vector4>,System.Exception&)
0x00000173755B8D5B (Mono JIT Code) UnityEngine.UIElements.UIR.UIRenderDevice:smile:rawChain (UnityEngine.UIElements.UIR.RenderChainCommand,UnityEngine.Rect,UnityEngine.Matrix4x4,UnityEngine.UIElements.PanelClearFlags,UnityEngine.Texture,UnityEngine.Texture,UnityEngine.Texture,single,Unity.Collections.NativeArray`1<UnityEngine.UIElements.UIR.Transform3x4>,Unity.Collections.NativeArray`1<UnityEngine.Vector4>,System.Exception&)
0x00000173755937EB (Mono JIT Code) UnityEngine.UIElements.UIR.RenderChain:Render (UnityEngine.Rect,UnityEngine.Matrix4x4,UnityEngine.UIElements.PanelClearFlags)
0x000001737558F2FB (Mono JIT Code) UnityEngine.UIElements.UIRRepaintUpdater:smile:rawChain (UnityEngine.Rect,UnityEngine.Matrix4x4)
0x000001737558E2DE (Mono JIT Code) UnityEngine.UIElements.UIRRepaintUpdater:Update ()
0x000001732CA540DA (Mono JIT Code) UnityEngine.UIElements.VisualTreeUpdater:UpdateVisualTreePhase (UnityEngine.UIElements.VisualTreeUpdatePhase)
0x000001737558C5A3 (Mono JIT Code) UnityEngine.UIElements.Panel:UpdateForRepaint ()
0x000001737558BC8B (Mono JIT Code) UnityEngine.UIElements.Panel:Repaint (UnityEngine.Event)
0x000001732CA52EE1 (Mono JIT Code) UnityEngine.UIElements.UIElementsUtility:smile:oDispatch (UnityEngine.UIElements.BaseVisualElementPanel)
0x000001732CA5289B (Mono JIT Code) UnityEngine.UIElements.UIElementsUtility:ProcessEvent (int,intptr)
0x000001732CA520E3 (Mono JIT Code) UnityEngine.GUIUtility:ProcessEvent (int,intptr)

i get this crash error with unity 2019.4 LTS

Stack Trace of Crashed Thread 14372:
ERROR: SymGetSymFromAddr64, GetLastError: ‘Es wurde versucht, auf eine unzulässige Adresse zuzugreifen.’ (Address: 00000001415B39A7)
0x00000001415B39A7 (Unity) (function-name not available)
ERROR: SymGetSymFromAddr64, GetLastError: ‘Es wurde versucht, auf eine unzulässige Adresse zuzugreifen.’ (Address: 000000014159A84E)
0x000000014159A84E (Unity) (function-name not available)
0x00000000029584FB (UnityEditor) UnityEditor.HandleUtility.IntersectRayMesh_Injected()

I’ve only recently realized that there is already an exposed API for raycasting into scene since Unity 5: HandleUtility.RaySnap.

P.S. My mistake. RaySnap doesn’t work with scene geometry without colliders so it isn’t really a solution to the original problem.

1 Like

Nice find, thank you for sharing. Have you tested it with SceneView.duringSceneGui? I’ve had some problems with raycasting from there.

With the original solution, I’ve had this issue on 2019 LTS. Skipping Layout and Repaint events as suggested has worked for me (it has also worked well inside SceneView.duringSceneGui on Windows). Here’s the code for reference:

private static readonly MethodInfo intersectRayMeshMethod = typeof( HandleUtility ).GetMethod( "IntersectRayMesh", BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Static );
private static GameObject lastGameObjectUnderCursor;

private void OnSceneGUI()
{
    if( RaycastAgainstScene( out RaycastHit hit ) )
    {
        // Use RaycastHit here
        Handles.ArrowHandleCap( 0, hit.point, Quaternion.LookRotation( hit.normal ), HandleUtility.GetHandleSize( hit.point ), Event.current.type );
    }
}

public static bool RaycastAgainstScene( out RaycastHit hit )
{
    Ray ray = HandleUtility.GUIPointToWorldRay( Event.current.mousePosition );

    // First, try raycasting against scene geometry with or without colliders (it doesn't matter)
    // Credit: https://forum.unity.com/threads/editor-raycast-against-scene-meshes-without-collider-editor-select-object-using-gui-coordinate.485502
    GameObject gameObjectUnderCursor;
    switch( Event.current.type )
    {
        // HandleUtility.PickGameObject doesn't work with some EventTypes in OnSceneGUI
        case EventType.Layout:
        case EventType.Repaint:
        case EventType.ExecuteCommand: gameObjectUnderCursor = lastGameObjectUnderCursor; break;
        default: gameObjectUnderCursor = HandleUtility.PickGameObject( Event.current.mousePosition, false ); break;
    }

    if( gameObjectUnderCursor )
    {
        Mesh meshUnderCursor = null;
        if( gameObjectUnderCursor.TryGetComponent( out MeshFilter meshFilter ) )
            meshUnderCursor = meshFilter.sharedMesh;
        if( !meshUnderCursor && gameObjectUnderCursor.TryGetComponent( out SkinnedMeshRenderer skinnedMeshRenderer ) )
            meshUnderCursor = skinnedMeshRenderer.sharedMesh;

        if( meshUnderCursor )
        {
            // Remember this GameObject so that it can be used inside problematic EventTypes, as well
            lastGameObjectUnderCursor = gameObjectUnderCursor;

            object[] rayMeshParameters = new object[] { ray, meshUnderCursor, gameObjectUnderCursor.transform.localToWorldMatrix, null };
            if( (bool) intersectRayMeshMethod.Invoke( null, rayMeshParameters ) )
            {
                hit = (RaycastHit) rayMeshParameters[3];
                return true;
            }
        }
        else
            lastGameObjectUnderCursor = null;
    }

    // Raycast against scene geometry with colliders
    object raycastResult = HandleUtility.RaySnap( ray );
    if( raycastResult != null && raycastResult is RaycastHit )
    {
        hit = (RaycastHit) raycastResult;
        return true;
    }

    hit = new RaycastHit();
    return false;
}
3 Likes

Thank you for sharing!
Back when I faced the reflections error I used a per triangle raycast. It works on any mesh and performance is decent (though I would prefer your native solution now).

For the interested here is the code (maybe useful for selecting single triangles):

/// <summary>
/// Returns the distance to the mesh from the raycast origin. Returns float.NaN if no hit was found.
/// </summary>
/// <param name="ray"></param>
/// <param name="objectWithMeshFilter"></param>
/// <param name="maxDistance"></param>
/// <returns>Distance as float or float.NaN</returns>
private static float RayCastMesh(Ray ray, GameObject objectWithMeshFilter, float maxDistance = 1000f)
{
    if (objectWithMeshFilter != null)
    {
        float distance = float.NaN;
        // check all triangles for hit with raycast
        var meshFilter = objectWithMeshFilter.GetComponent<MeshFilter>();
        if (meshFilter != null)
        {
            int[] triangles = meshFilter.sharedMesh.triangles;
            Vector3[] vertices = meshFilter.sharedMesh.vertices;
            int a, b, c;
            for (int i = 0; i < triangles.Length; i = i + 3)
            {
                a = triangles[i];
                b = triangles[i + 1];
                c = triangles[i + 2];
                /*
                Debug.DrawLine(meshFilter.transform.TransformPoint(vertices[a]), meshFilter.transform.TransformPoint(vertices[b]), Color.red, 3.0f);
                Debug.DrawLine(meshFilter.transform.TransformPoint(vertices[b]), meshFilter.transform.TransformPoint(vertices[c]), Color.red, 3.0f);
                Debug.DrawLine(meshFilter.transform.TransformPoint(vertices[c]), meshFilter.transform.TransformPoint(vertices[a]), Color.red, 3.0f); //*/
                distance = IntersectRayTriangle(
                    ray,
                    meshFilter.transform.TransformPoint(vertices[a]),
                    meshFilter.transform.TransformPoint(vertices[b]),
                    meshFilter.transform.TransformPoint(vertices[c]));
                if (!float.IsNaN(distance))
                {
                    break;
                }
            }
        }
        if (float.IsNaN(distance) == false)
        {
            if (distance < maxDistance)
            {
                return distance;
            }
        }
    }

    return float.NaN;
}

const float kEpsilon = 0.000001f;

/// <summary>
/// Thanks to: https://answers.unity.com/questions/861719/a-fast-triangle-triangle-intersection-algorithm-fo.html
/// Ray-versus-triangle intersection test suitable for ray-tracing etc.
/// Port of Möller–Trumbore algorithm c++ version from:
/// https://en.wikipedia.org/wiki/Möller–Trumbore_intersection_algorithm
/// </summary>
/// <returns><c>The distance along the ray to the intersection</c> if one exists, <c>NaN</c> if one does not.</returns>
/// <param name="ray">the ray</param>
/// <param name="v0">A vertex 0 of the triangle.</param>
/// <param name="v1">A vertex 1 of the triangle.</param>
/// <param name="v2">A vertex 2 of the triangle.</param>
public static float IntersectRayTriangle(Ray ray, Vector3 v0, Vector3 v1, Vector3 v2)
{
    // edges from v1 & v2 to v0. 
    Vector3 e1 = v1 - v0;
    Vector3 e2 = v2 - v0;

    Vector3 h = Vector3.Cross(ray.direction, e2);
    float a = Vector3.Dot(e1, h);
    if ((a > -kEpsilon) && (a < kEpsilon))
    {
        return float.NaN;
    }

    float f = 1.0f / a;

    Vector3 s = ray.origin - v0;
    float u = f * Vector3.Dot(s, h);
    if ((u < 0.0f) || (u > 1.0f))
    {
        return float.NaN;
    }

    Vector3 q = Vector3.Cross(s, e1);
    float v = f * Vector3.Dot(ray.direction, q);
    if ((v < 0.0f) || (u + v > 1.0f))
    {
        return float.NaN;
    }

    float t = f * Vector3.Dot(e2, q);
    if (t > kEpsilon)
    {
        return t;
    }
    else
    {
        return float.NaN;
    }
}
1 Like

Does this work in the Game window as well ?

HandleUtility.PickGameObject doesn’t work inside Game window.

1 Like

I don’t think it works on the game scene while editing (it’s hard to get any events from the game view if you’re not in play mode). If you rather mean “in game view at run time” then I don’t see why the manual (shared mesh + triangle check) should not work (as long as the mesh is readable). Though I have not tested it. Just give it a try.

1 Like

Throwing up this error - The name ‘kEpsilon’ does not exist in the current context

Ah sry, I just ripped it out of my code. It’s just a float comparison delta value (added it above).

const float kEpsilon = 0.000001f;
1 Like

@ : Thanks a lot for the code. It works fine. :slight_smile:

1 Like

@geo_ : Can we modify this code to know if the Ray is hitting the Object or not ?

Have any suggestions on this?

That’s what “RayCastMesh” from above does. If it returns float.NaN then the game object was not hit.

1 Like

@geo_:Is there a way to find the closest point from the mesh. I am unable to decode the code.

If you can shed some light, that would be really grateful.

This is what I have done:

int[ ] triangles = meshFilter.sharedMesh.triangles;
Vector3[ ] vertices = meshFilter.sharedMesh.vertices;
int a, b, c;

Vector3[ ] worldPOS = new Vector3[vertices.Length];
for (int j = 0; j < vertices.Length; j++)
{
worldPOS[j] = objectWithMeshFilter.transform.TransformPoint(vertices[j]);

if (temp == 0)
{
temp = Vector3.Distance(worldPOS[j], Player.transform.position);
}
else if (temp >Vector3.Distance(worldPOS[j], Player.transform.position))
{
temp = Vector3.Distance(worldPOS[j], Player.transform.position);
temp_ = worldPOS[j];
Debug.Log("Distance: "+ temp);

}
}

Can someone from Unity please just expose whatever Middle-Mouse click does in the SceneView? We need a way to get the result of a ray-cast into the SceneView without requiring that everything have colliders. Trying to update visible and collision meshes at the same time as they’re modified with our mesh editing tool is just too slow.