Custom Raycaster.

Hello,

I’m working on a custom Raycaster that will sort the raycast results in priority based on their layers and so long as their distance is not too far from the object that is closest to the camera.

Basically, I have a character that has an EventTrigger, there are some obstacles that also have an EventTrigger but if the character is relatively close I want the input to go to the character and not the obstacle. Because the regular PhysicsRaycaster seems to sort the the RaycastHit based on distance, the closest one that is eligible for interaction gets the message.

So I looked at the code in BitBucket and just followed the flow but even though my final RaycastResult list is sorted (according to my requirements) it doesn’t matter what I say should be on top the closest one still gets priority and gets the PointerEvents.

I’ve attached a screenshot, as you can see in the console Player is at the front (which is what I want) but if you look at the EventSystem “Cube” (which has a lower priority but is ‘technically’ closer) is receiving the PointerEvent data.

Any help would be great thanks!

Code

using UnityEngine;
using UnityEngine.EventSystems;
using System.Collections.Generic;

[RequireComponent(typeof(Camera))]
public class PriorityRaycaster : PhysicsRaycaster {
    public List<LayerMask> maskPriority = new List<LayerMask>();
    public float deltaDistance = 0.5F;
    public List<RaycastHit> sortedCache = new List<RaycastHit>();

    public override void Raycast(PointerEventData eventData, List<RaycastResult> resultAppendList) {
        if (this.eventCamera == null) return;

        this.sortedCache.Clear();

        var ray = this.eventCamera.ScreenPointToRay(eventData.position);

        var distance = this.eventCamera.farClipPlane - this.eventCamera.nearClipPlane;

        var hits = Physics.RaycastAll(ray, distance, this.finalEventMask);

        if(hits.Length > 1) {
            // sort first based on distance (lower is better)
            System.Array.Sort(hits, (a, b) => a.distance.CompareTo(b.distance));
        }

        if(hits.Length != 0) {
            // What is smallest distance
            // this will be our bounds to make sure we don't give
            // priority to an object that is really far away.
            var closest = hits[0].distance;

            for(int i = 0 ; i < hits.Length ; ++i) {
                this.SortByLayerMask(hits[i], this.sortedCache, closest);
            }

            eventData.worldPosition = this.sortedCache[0].point;
            eventData.worldNormal = this.sortedCache[0].normal;

            // For debugging purposes...
            string print = "[";

            for(int i = 0 ; i < this.sortedCache.Count ; ++i) {
                var result = new RaycastResult {
                    gameObject = this.sortedCache[i].collider.gameObject,
                    module = this,
                    distance = this.sortedCache[i].distance,
                    index = resultAppendList.Count
                };

                resultAppendList.Add(result);

                print += result.gameObject.name + " ";
            }

            print += "]";

            // Print out the final results in order of LayerMask priority.
            // The higher in the list in the inspector the higher the priority.
            // Index = 0 has highest priority.
            Debug.Log(print);
        }
    }

    protected virtual void SortByLayerMask(RaycastHit hit, List<RaycastHit> sortedList, float closestDistance) {
        // Find the index in the maskPriority list (lower is higher priority)
        var hitIndex = maskPriority.FindIndex(i => PriorityRaycaster.IsInLayer(hit.collider.gameObject.layer, i));

        //Debug.Log(
            //string.Format(
            //    "HitIndex {0} Name {1} Value {2}",
            //    hitIndex,
            //    hit.collider.name,
            //    hit.transform.gameObject.layer
            //    )
            //);
      
        // Is there more than 1 item in the list?
        if(sortedList.Count >= 1) {
            // Find the index in the maskPriority list (lower is higher priority)
            var front = maskPriority.FindIndex(i => PriorityRaycaster.IsInLayer(sortedList[0].collider.gameObject.layer, i));

            // If the hitIndex has a higher priority
            // and the distance from the closest hit to the camera is less than deltaDistance
            // insert it to the front.
            if(hitIndex < front && Mathf.Abs(hit.distance - closestDistance) <= this.deltaDistance) {
                sortedList.Insert(0, hit);
            } else {
                var insertIndex = -1;

                for(int i = 0 ; i < sortedList.Count ; ++i) {
                    var maskIndex = maskPriority.FindIndex(mask => PriorityRaycaster.IsInLayer(sortedList[i].collider.gameObject.layer, mask));

                    // Find the next index to insert this RaycastHit
                    if(hitIndex < maskIndex && Mathf.Abs(hit.distance - closestDistance) <= this.deltaDistance) {
                        insertIndex = i;
                        break;
                    }
                }

                // Insert into the back (lower priority)
                if (insertIndex == -1) sortedList.Add(hit);
                // Insert at the next best location.
                else sortedList.Insert(insertIndex, hit);
            }
        } else {
            // just insert it.
            sortedList.Add(hit);
        }
    }

    /// <summary>
    /// Checks if <paramref name="layer"/> is inside <paramref name="mask"/>.
    /// </summary>
    /// <returns><c>true</c> if the layer is in the mask and <c>false</c> otherwise.</returns>
    /// <param name="layer">Layer.</param>
    /// <param name="mask">Mask.</param>
    static bool IsInLayer(int layer, LayerMask mask) {
        return mask == (mask | (1 << layer));
    }
}

Hey, I know this post is old, but did you find a soloution? I am struggeling with the same thing

Well, your custom raycaster is just one that may apply at the same time. Those raycasters are used by the EventSystem to collect multiple results which are then sorted based on the default sorter. Unfortunately this sorter is hardwired and can’t be replaced that easily. So your best bet would be to see on what properties it actually sorts the results and just set those properties in your own raycaster so the sorter would sort them in the way you want.

Thank you for the reply!
Meanwhile I managed to make it work. My Raycaster does 2D raycasts an looks for the objects 2D renderers and adds to the append-list whatever object is rendered frontmost (by comparing sorting layer id and sorting order). The Problem that I have now is, that it doesn’t work with sorting groups. But that is a whole new topic.