RaycastHit.triangleIndex choosing wrong triangle

Unity Engine
1

I’ve got some editor-script code where you mouseclick in the scene view, and it uses RaycastHit.triangleIndex to highlight the triangle you hit. But in a new scene, it’s often choosing the wrong triangle - usually a triangle connected to the one clicked. Sometimes it even has two triangles where clicking on one always selects the other. Any ideas what could cause this?

The one special thing about this scene (that I wouldn’t think would be the problem), the objects I’m selecting are all just one giant single mesh, with no submeshes, spread across multiple separate (non-contiguous) parts. Note however that the highlighted triangle has always been on the same part I clicked on, just the wrong triangle.

Code for the mouse click:

    static void PathDebugBoundariesGUI(SceneView sceneView)
    {
        Event cur = Event.current;
        if (cur.type == EventType.MouseDown && cur.button != 2)
        {
            RaycastHit[] hits;
            Ray clickRay = HandleUtility.GUIPointToWorldRay(cur.mousePosition);

            hits = Physics.RaycastAll (clickRay);
            if (hits.Length > 0)
            {
                foreach (RaycastHit hit in hits)
                {
                    if (hit.collider.name.ToLower ().Contains ("path"))
                    {
                        GetTriangleVerticesFromRaycast (hit);
                        SceneView.RepaintAll ();
                        break;
                    }
                }
            }
        }
    }

Code that gets the triangle from the mesh:

    static void GetTriangleVerticesFromRaycast(RaycastHit hit)
    {
        Mesh mesh = (hit.collider as MeshCollider).sharedMesh;
        if (mesh != null)
        {
            int[] triangles = mesh.triangles;
            Vector3[] vertices = mesh.vertices;
            Transform hitTrans = hit.collider.transform;
            patherTool_.firstObject_[0] = hitTrans.TransformPoint (vertices [triangles [hit.triangleIndex * 3 + 0]]);
            patherTool_.firstObject_[1] = hitTrans.TransformPoint (vertices [triangles [hit.triangleIndex * 3 + 1]]);
            patherTool_.firstObject_[2] = hitTrans.TransformPoint (vertices [triangles [hit.triangleIndex * 3 + 2]]);
        }
    }

Code that highlights the triangle:

        if (firstObject_[0] != Vector3.zero)
        {
            Debug.DrawLine(firstObject_[0], firstObject_[1], Color.magenta);
            Debug.DrawLine(firstObject_[1], firstObject_[2], Color.magenta);
            Debug.DrawLine(firstObject_[2], firstObject_[0], Color.magenta);
        }
2

Try drawing some debug lines to see where the ray starts and goes to (hit.point) with Debug.DrawRay/Line?

Also in the event.mouseposition y is inverted, compared to input.mouseposition, not sure if GUIPointToWorldRay takes that into account already…

3

Unfortunately, I’ve done that; it has nothing to do with the Ray itself being wrong. Plus I’m also doing other raycasts against the objects, not from the mouse position but from code, and those are getting the wrong results as well. What’s wrong is the triangleIndex being returned.

Here’s an image showing what I’m seeing - the little selected sphere is being drawn at the RaycastHit.point, the triangle it’s returning is highlighted in blue (from the code above, though blue instead of magenta).

447×718 55.8 KB

Let me tell you what I’m actually doing - I have line meshes forming roads, and I’m trying to create a path along the center of the road. To do so, my editor script first has you choose two triangles on lines on opposite sides of the street; the script finds the edge of the two triangles closest to each other, then gets the direction perpendicular to that edge, gets the distance from one triangle edge to the other, uses that information to find the exact center of the road. Then it moves down the center of that road a bit perpendicular to one of the edges, and starts checking to the sides casting rays again until it finds the road lines, gets the first triangles it hits, and repeats. So choosing the correct triangle is vastly important.

This scene has many of the lines combined into one big mesh with no submeshes, which is the only thing I can think of that could be causing the problem. But I’m not sure why that would affect it, or what to do to fix it other than having an artist separate all the lines into unique meshes.

4

@Dreamback : Did you find a solution for your problem?