RTS selection with frustum

So, pretty standard task for RTS game: select multiple units via mouse.

I use well-known way: call ScreenPointToRay for starting and ending points, calculate rectangle with World coordinates and call Physics.OverlapBox with given size and camera rotation.

The problem begins with selection for units that are closer to left/right edges, because Box is not the right figure. It must be some kind of frustum with not-parallel top and bottom planes (terrain is horizontal, camera rotated 45deg to terrain).

In other words, I need to make kind of 3D collider from camera selection rect to projected rect on terrain and than find object that collided with it.

I know there is a CalculateFrustumPlanes for getting 6 planes, corresponds to camera viewport, but I need narrower figure (only selected part of camera viewport) and I haven’t found any method to find Overlaps within planes intersection.

This suggestion doesn’t factor in Physics.OverlapBox(), and uses just the center point of each unit, but it should definitely be adaptable as needed.

Let’s start by plotting out efficiency at the beginning. It may seem like a silly thing to do, but for a Real Time Strategy game, where you can expect most units to not be visible on screen at a time, this should be able to help significantly overall.

First, let’s combine [MonoBehaviour/Renderer.OnBecameVisible()][1] with a handy custom script extension, [Renderer.IsVisibleFrom(Camera)][2].

void OnBecameVisible()
{
	// Simplifying a few variables that are presumed to be kept track of
	
	// if the player's main camera sees the unit
	// and the player owns that unit...
	if(renderer.IsVisibleFrom(playerCamera) && unit.owner == player)
	{
		// Add the unit to a List<Transform> of friendly/owned units currently in view
		player.visibleUnits.Add(transform);
	}
}

void OnBecameInvisible()
{
	if(player.visibleUnits.Contains(transform))
	{
		player.visibleUnits.Remove(transform);
	}
}

With this out of the way, there are now far fewer units to work with at a time when determining your selection bounds. From here, after you’ve created your selection box in your viewport, you can run through the on-screen unit list and convert the units’ positions to viewport coordinates:

// When the mouse is released after creating the selection box...
if(selectionBoxFinalized)
{
	for(int i = 0; i < player.visibleUnits.Count; i++)
	{
		// Screen point, viewport point, etc.
		if(selectionBoxRect.Contains(playerCamera.WorldToViewportPoint(player.visibleUnits*.position)))*
  •  {*
    

_ AddToSelection(player.visibleUnits*);_
_
}_
_
}_
_
}_
_


*_
Even if this doesn’t suit your needs exactly, hopefully this can help get you pointed in a good direction.

Edit: Fixed typo
_[1]: https://docs.unity3d.com/ScriptReference/Renderer.OnBecameVisible.html*_
_
[2]: https://wiki.unity3d.com/index.php/IsVisibleFrom*_

So, what I have now.
Camera is inside CameraRig object

CameraRig - pos (0, 0, 0) rot (0, 0, 0)
===> Camera - pos (0, 20, -40) rot (45, 0, 0)

I visualized front face rect, as you can see if there is no rotation (or rotation by 180deg) - all ok
[172611-снимок-экрана-2020-12-16-в-044441.jpg|172611]

But when I rotate CameraRig (rotation Y) I get same orientation for that rect (even with correct position).

[172612-снимок-экрана-2020-12-16-в-044538.jpg|172612]

Here is some code

// selectionStartMouse == Input.mousePosition at start
// selectionEndMouse == Input.mousePosition now

var frontRect = GetScreenRect(selectionStartMouse, selectionEndMouse);

var frontBottomLeft = GetMouseToWorldPoint(new Vector3(frontRect.xMin, frontRect.yMin));
var frontTopRight = GetMouseToWorldPoint(new Vector3(frontRect.xMax, frontRect.yMax));

// I know that next two lines are incorrect, cause calculation of last 2 corners of rect will work that way only without rotation, but the main problem with first two from ScreenToWorldPoint
var frontTopLeft = frontTopRight.SetX(frontBottomLeft.x); // SetX == replace x coord
var frontBottomRight = frontBottomLeft.SetX(frontTopRight.x);

    public static Rect GetScreenRect(Vector3 screenPosition1, Vector3 screenPosition2)
    {
        var topLeft = Vector3.Min(screenPosition1, screenPosition2);
        var bottomRight = Vector3.Max(screenPosition1, screenPosition2);
        
        return Rect.MinMaxRect(topLeft.x, topLeft.y, bottomRight.x, bottomRight.y);
    }

    public static Vector3 GetMouseToWorldPoint(Vector3 coords)
    {
        coords.z = cam.nearClipPlane + 0.5f; // move away a bit from viewer

        return GetCamera().ScreenToWorldPoint(coord);
    }

I’ve searched and seen this question asked and answered at least a dozen times, but every single answer seems to make unreasonable simplifying assumptions about how the petitioner is making their game, and typically totally disregard a lot of edge cases which can not be ignored if you want to make a functional game. So, for all future internet searchers trying to find the answer to this question: here is code that will actually create a frustum from a screen-space Rect by determining the frustum corners in world space and populating a set of world-space planes.

public static class CameraExtensionsRts
{
    const Camera.MonoOrStereoscopicEye k_CameraMode = Camera.MonoOrStereoscopicEye.Mono;

    public static void CalculateSubFrustum(this Camera camera, Rect viewport, Plane[] frustumPlanes,
        Camera.MonoOrStereoscopicEye cameraMode = k_CameraMode)
        => camera.CalculateSubFrustum(viewport, frustumPlanes, camera.nearClipPlane, camera.farClipPlane, cameraMode);

    public static void CalculateSubFrustum(this Camera camera,
        Rect screen, Plane[] frustumPlanes, float near, float far,
        Camera.MonoOrStereoscopicEye cameraMode = k_CameraMode)
    {
        // NOTE: We're assuming the Rect passed in was read directly from mouse (pixel) coordinates, whereas the Rect
        //       CalculateFrustumCorners uses below takes viewport (normalized to [0-1]) coordinates
        var min = (Vector2)camera.ScreenToViewportPoint(screen.min);
        var max = (Vector2)camera.ScreenToViewportPoint(screen.max);
        var viewport = Rect.MinMaxRect(min.x, min.y,  max.x, max.y);
        var nearCornersLocalSpace = new Vector3[4];
        var farCornersLocalSpace = new Vector3[4];
        // NOTE: This isn't in the documentation anywhere but it looks like output is in the camera's local space
        //       and always starts with the bottom left corner and winds clockwise
        camera.CalculateFrustumCorners(viewport, near, cameraMode, nearCornersLocalSpace);
        camera.CalculateFrustumCorners(viewport, far, cameraMode, farCornersLocalSpace);
        // Transform local coordinates to world and concatenate into single array, ordered as follows
        // indices 0-3 => near plane || indices 4-7 => far plane
        // 0, 1, 2, 3 => bottom left, top left, top right, and bottom right, respectively, looking out from camera
        // 4, 5, 6, 7 keep the same respective ordering
        var frustumCornersIn = nearCornersLocalSpace
            .Select(camera.transform.TransformPoint)
            .Concat(farCornersLocalSpace
                .Select(camera.transform.TransformPoint))
            .ToArray();
        // Populate the frustum planes according to the CalculateFrustumPlanes API
        // (which for some reason doesn't take a viewport argument)
        // https://docs.unity3d.com/ScriptReference/GeometryUtility.CalculateFrustumPlanes.html
        // Left Plane
        frustumPlanes[0].Set3Points(frustumCornersIn[0], frustumCornersIn[1], frustumCornersIn[5]);
        // Right Plane
        frustumPlanes[1].Set3Points(frustumCornersIn[2], frustumCornersIn[3], frustumCornersIn[7]);
        // Bottom Plane
        frustumPlanes[2].Set3Points(frustumCornersIn[0], frustumCornersIn[4], frustumCornersIn[7]);
        // Top Plane
        frustumPlanes[3].Set3Points(frustumCornersIn[1], frustumCornersIn[2], frustumCornersIn[6]);
        // Near Plane
        frustumPlanes[4].Set3Points(frustumCornersIn[2], frustumCornersIn[1], frustumCornersIn[0]);
        // Far Plane
        frustumPlanes[5].Set3Points(frustumCornersIn[4], frustumCornersIn[5], frustumCornersIn[6]);
    }
}

Then, inside your input handling class you would need something like this.

public class YourSelectionManager : MonoBehaviour
{
    bool m_IsBoxSelecting;
    // NOTE: Might want to use Renderer visibility events to keep this list short
    // https://docs.unity3d.com/ScriptReference/Renderer.OnBecameVisible.html
    List<GameObject> m_SelectableUnits;

    // Start this coroutine when a click-n-drag starts -- pass it the currently active Camera
    IEnumerator DoBoxSelect(Camera camera)
    {
        m_IsBoxSelecting = true;
        var start = GetPointerPositionScreenSpace();
        // NOTE: You'll need to set m_IsBoxSelecting to false from whatever is monitoring the user input
        yield return new WaitUntil(() => !m_IsBoxSelecting);
        var end = GetPointerPositionScreenSpace();

        var min = Vector2.Min(start, end);
        var max = Vector2.Max(start, end);
        var box = Rect.MinMaxRect(min.x, min.y, max.x, max.y);
        // Box was not dragged out - just select the unit under the cursor
        if (box.size == Vector2.zero)
            DoClickSelect();
        // If sides are too small, we'll create infinite planes when calculating the frustum, so clamp up a bit
        const float sideLengthMin = 1e-2f;
        if (box.width < sideLengthMin)
            box.width = sideLengthMin;
        if (box.height < sideLengthMin)
            box.height = sideLengthMin;

        var frustumPlanes = new Plane[6];
        camera.CalculateSubFrustum(box, frustumPlanes);

        foreach (var unit in m_SelectableUnits)
        {
            // Get whatever bounds are relevant to this unit - could be Renderer or Collider Bounds, just
            // do NOT use MeshFilter Bounds (those are in local space)
            var boundingBoxes = GetBounds(unit);
            // If you are always using one Bounds per unit, this can be simplified by 
            // not returning/iterating on a List
            if (boundingBoxes.Any(bounds => GeometryUtility.TestPlanesAABB(frustumPlanes, bounds)))
                AddToSelection(unit);
        }
    }


    // TODO: You implement these...
    Vector2 GetPointerPositionScreenSpace() => throw new System.NotImplementedException();
    void DoClickSelect() => throw new System.NotImplementedException();
    void AddToSelection(GameObject selected) => throw new System.NotImplementedException();
    List<Bounds> GetBounds(GameObject unit) => throw new System.NotImplementedException();
}

This will take the box a player draws in screen space and turn it into a world-space frustum that you can test your Bounds against using a built-in Unity API. A few important things to note:

  1. This frustum doesn’t do occlusion checking by itself, it will return positive if you pass it a bounds that is within the frustum even if those bounds are not visible because they’re behind something
  2. This ONLY works for cameras rendering Perspective, not Othographic. There are plenty of solutions to be found for Orthographic cameras as you’re safe to simply extend a big rectangle out of the camera into your scene.
  3. World-space Axis-Aligned Bounding Boxes (AABB) often extend outside the visual boundaries of your object, meaning that without additional constraints you will sometimes select objects when the box-select overlaps the bounding box, but not necessarily the rendered object. If you are using collider bounds and your colliders are well-constrained this shouldn’t be much of an issue.