Bounding box around the shadow of an object

Wattosan · September 1, 2017, 4:38am

Hello,

Is it possible to draw an outline/bounding box around the object’s shadow only?

I found this thread, which explains how to draw a bounding box around an object. How to display a rectangle around a player? - Questions & Answers - Unity Discussions.

However, is it possible to draw one around the shadow of an object?

This is what I’d like to achieve:

I need this to make the a screenshot of only the shadow. And the screenshot’s width and height has to be as minimum as possible. The shadow must fit in perfectly.

Thanks

jvo3dc · September 1, 2017, 11:56am

It’s always possible, but it might be very tricky.

Is the surface the shadow casts on always flat? In that case it’s very easy, just make a copy of the model and project it onto the plane. Then draw a bounding box around that object.

In the case the surface is not always flat, it becomes a very different exercise.

Wattosan · September 3, 2017, 3:42pm

Yes, it is always flat. However, I quite don’t understand the solution you propose. You suggest I make a copy of the object that casts the shadow (a box for example) and somehow project it onto the plane. How does one project it onto the plane? (and yes, the shadow is always cast on a plane)

Thank you!

PS. Another, a more difficult solution, which I thought of was to create a setup of only one directional light. I will know the angle of the light. I also know the position of the object that casts the shadow. With this information, I should be able to calculate the length (size) of the shadow: its final starting and end position.

jvo3dc · September 4, 2017, 9:13am

Well, just a little vector math can do the projection on a plane. Can be done for any plane normal, but in this case I think it’s save to assume the plane normal is just positive y.

Vector3 sunDir; // Sun direction
Vector3 vertex; // Vertex xyz
float planeY; // Y value of plane
vertex += sunDir * ((vertex.y - planeY) / sunDir.y); // Projection

This also assumes a directional light of course.

Wattosan · September 4, 2017, 12:35pm

I tried this solution and came up with this code:

[SerializeField] Transform sunTransform;
[SerializeField] Transform planeTransform;
[SerializeField] MeshFilter mf;
   
void Update () {
        Vector3 sunDir =sunTransform.localRotation.eulerAngles;
        for (int i=0; i< mf.mesh.vertexCount; i++)
        {
            Vector3 vertex = mf.mesh.vertices[i];
            float planeY = planeTransform.position.y; // Y value of plane
            vertex += sunDir * ((vertex.y - planeY) / sunDir.y); // Projection
        }
    }

This however, froze Unity due to the high amount of vertices. And secondly, when I tried it with a cube (to prevent freezing), nothing really happened.

jvo3dc · September 4, 2017, 5:41pm

No, nothing will really happen that way. You need to actually maintain the list of vertices and set it back on the mesh. Things are pass by copy in C#, not pass by reference.

bgolus · September 5, 2017, 4:22am

For what you want I would suggest projecting only the bounding box of your object onto the plane. This kind of mass calculation is what GPUs eat for breakfast, far more complex math is being done to render it on screen every frame. CPUs aren’t quite as good at this, so even this simple projection is going to be way too slow to do, as you experienced.

Basic steps:

Get local mesh bounds.
Transform corners into world space.
Project corners onto plane.
Transform from world space into screen space.
Get min & max bounds of those transformed corners in screen space.
Draw bounds.
…
Profit!

I suggest starting with the local mesh bounds rather than the world space renderer bounds as it should result in a tighter on screen bounds than using the renderer bounds. The 8 additional local to world transforms shouldn’t be a problem for performance.

bgolus · September 5, 2017, 4:24am

Also, because it’ll bite you of you don’t, you need to use Unity - Scripting API: Renderer.localToWorldMatrix to transform the local mesh bounds into world space, not the Transform local to world function.

Wattosan · September 5, 2017, 8:20am

@bgolus
Hey, so I tried putting the steps you wrote into code and came up with this:
ProjectScript

using UnityEngine;

public class ModelProjector : MonoBehaviour {

    [SerializeField] Camera cam;
    [SerializeField] Transform sunTransform;
    [SerializeField] Transform planeTransform;
    [SerializeField] MeshFilter mf;
    [SerializeField] MeshRenderer mr;

    private Vector3[] pts = new Vector3[8];

    void OnGUI() {

        // 1. Get local mesh bounds
        // Bounds mfBounds = mf.mesh.bounds;
        Bounds mfBounds = mr.bounds;

        // 2. Transform corners into world space
        // All 8 vertices of the bounds
        pts[0] = new Vector3(mfBounds.center.x + mfBounds.extents.x, mfBounds.center.y + mfBounds.extents.y, mfBounds.center.z + mfBounds.extents.z);
        pts[1] = new Vector3(mfBounds.center.x + mfBounds.extents.x, mfBounds.center.y + mfBounds.extents.y, mfBounds.center.z - mfBounds.extents.z);
        pts[2] = new Vector3(mfBounds.center.x + mfBounds.extents.x, mfBounds.center.y - mfBounds.extents.y, mfBounds.center.z + mfBounds.extents.z);
        pts[3] = new Vector3(mfBounds.center.x + mfBounds.extents.x, mfBounds.center.y - mfBounds.extents.y, mfBounds.center.z - mfBounds.extents.z);
        pts[4] = new Vector3(mfBounds.center.x - mfBounds.extents.x, mfBounds.center.y + mfBounds.extents.y, mfBounds.center.z + mfBounds.extents.z);
        pts[5] = new Vector3(mfBounds.center.x - mfBounds.extents.x, mfBounds.center.y + mfBounds.extents.y, mfBounds.center.z - mfBounds.extents.z);
        pts[6] = new Vector3(mfBounds.center.x - mfBounds.extents.x, mfBounds.center.y - mfBounds.extents.y, mfBounds.center.z + mfBounds.extents.z);
        pts[7] = new Vector3(mfBounds.center.x - mfBounds.extents.x, mfBounds.center.y - mfBounds.extents.y, mfBounds.center.z - mfBounds.extents.z);


        // 3. Project corners onto plane
        Vector3 sunDir = sunTransform.localRotation.eulerAngles;
        for (int i=0; i < pts.Length; i++)
        {
       
            Vector3 cornerPoint = pts[i];
            float planeY = planeTransform.position.y; // Y value of plane
            pts[i] += sunDir * ((cornerPoint.y - planeY) / sunDir.y); // Projection
       
            // Vector3.ProjectOnPlane(pts[i], planeTransform.position.normalized);
        }

        // 4. Transform from world space into screen space
        for (int i = 0; i < pts.Length; i++)
        {
            Vector3 cornerPoint = pts[i];
            pts[i] = cam.WorldToScreenPoint(cornerPoint);
        }

        //Get them in GUI space
        for (int i = 0; i < pts.Length; i++) pts[i].y = Screen.height - pts[i].y;

        // 5. Get min & max bounds of those transformed corners in screen space.
        Vector3 min = pts[0];
        Vector3 max = pts[0];
        for (int i = 1; i < pts.Length; i++)
        {
            min = Vector3.Min(min, pts[i]);
            max = Vector3.Max(max, pts[i]);
        }

        // 6. Draw bounds
        Rect r = Rect.MinMaxRect(min.x, min.y, max.x, max.y);
        GUI.Box(r, "Shadow");

        // 7. ...

        // 8. Where are you, Mr. Profit?
    }
}

However, I am getting faulty results:

Any idea what I might be doing wrong? Perhaps, it’s the reason I am not using Renderer.localToWorldMatrix. But how do I use it? Even if I do get the 4x4 matrix from the bounds, how do I proceed with it?

It also does not seem to matter if I use MeshFilter.mesh.bounds or MeshRenderer.bounds.
I think the error is at step 3. Projecting. I tried to use the solution proposed above and the built in Vector3.ProjectOnPlane(). Neither seemed to do the trick. @jvo3dc mentioned I need to set them back on the mesh. Is that the problem?

Cheers

bgolus · September 5, 2017, 8:48am

The renderer bounds are already in world space, so those are fine to use. At least for now to make sure the rest of the code is working properly. To use the localToWorldMatrix you just need to transform the corners using:

point = Renderer.localToWorldMatrix(point);
https://docs.unity3d.com/ScriptReference/Matrix4x4.MultiplyPoint.html

But only do this if you’re using the local mesh bounds, not when using the renderer bounds. Leave it as is for now.

Definitely don’t use this. It’s doing something very different than what you want. What you want is to find a position projected onto a plane at an arbitrary projection angle. That function only handles the case where the projection angle is orthogonal to the plane. Another way to describe that is it’s finding the nearest point on a plane. In your case where the plane is the flat ground it would act as if the light was directly above aimed straight down. In that case point.y = groundHeight would be just as accurate, and faster! But that’s not what you need.

Here’s the real problem:

Both @jvo3dc and I missed this in your previous post. The sunDir value should be the sun transform’s forward vector, not the Euler angles. And by forward vector I mean literally sunTransform.forward. I’m honestly surprised the result you’re getting using the Euler angles is even appearing on screen, let alone aligned with the box.

edit: One minor comment on the code above. You have float planeY = planeTransform.position.y; inside your loop. Do that outside the loop, as that value is calculated every time it’s accessed. Technically it gets cached after the first access now, but it’ll still be faster to do that just before the loop instead of reaccessing it every time.

Wattosan · September 5, 2017, 9:43am

I tried to do this. And the result which I got was this:

First of all, it seems the drawn box is flipped. And secondly, it seems to be too big in most of the cases.

bgolus · September 5, 2017, 7:59pm

Yep. Something isn’t going right. It’s not obvious to me where or why.

Maybe “/ **-**sunDir.y”

I would suggest logging the projected corners, or have them as a public variable so you can see then in the inspector, and make sure their y value is properly equal to your plane’s height.

Wattosan · September 7, 2017, 7:22pm

Yes, thank you! You are correct. I have a working solution for directional light now.
Any ideas how to make it work with point lights?

bgolus · September 7, 2017, 8:16pm

Sure. You just need to do the same math, but replace the “sunDir” with (cornerPoint - worldLightPos).normalized for each light. Same should work for spotlights, but that won’t handle the spotlight’s cone. That’s a lot more work.

Wattosan · September 7, 2017, 10:39pm

Hey, I tried this, and it does work for one pointlight. But having more than one makes the box be rendered incorrectly:

https://www.youtube.com/watch?v=Z58SK97ArLs

To the code, I added the functionality, which finds each point light in the scene at “Start()”. And then changed the projection code:
Code

List<Transform> pointLightTransforms = new List<Transform>();

    private void Start()
    {
        foreach (Light l in FindObjectsOfType<Light>())
        {
            if (l.type == LightType.Point)
            {
                pointLightTransforms.Add(l.transform);
            }
        }
    }

    void OnGUI() {

        //    .
        //    . Commented this out for forum post
        //    .

        // 3. Project corners onto plane
     
        float planeY = 0;
        for (int i=0; i < pts.Length; i++)
        {
            Vector3 cornerPoint = pts[i];
            // Vector3 sunDir = sunTransform.forward;
            foreach (Transform pointLightTransform in pointLightTransforms)
            {
                Vector3 pointLightDir = (cornerPoint - pointLightTransform.position).normalized;
                planeY = planeTransform.position.y; // Y value of plane
                pts[i] += -pointLightDir * ((cornerPoint.y - planeY) / pointLightDir.y); // Projection
            }
        }

Do I somehow need to add/combine all the pointLightTransform.position vectors before adding them to the corners? Or is there something else missing? Or somehow check, which point light creates the furthest shadows in each of the 4 directions? (top, bottom, left and right).

bgolus · September 7, 2017, 11:17pm

You can’t add the projections together. You need to calculate the on-screen points for each light’s shadow separately. If you have 3 lights instead of 8 points to track you now have 24 points to track and find the on-screen position of.

I think the most efficient way would be to do the full loop of corners to on screen min-max for each light and keep track of the largest min and max on screen bounds.

Vector3 minBounds = new Vector3(Mathf.Infinity, Mathf.Infinity, Mathf.Infinity);
Vector3 maxBounds = new Vector3(Mathf.NegativeInfinity, Mathf.NegativeInfinity, Mathf.NegativeInfinity);
foreach light
  foreach point
    Vector3 tempPos = // project to plane w/ if for directional vs other
    tempPos = // transform to screen pos
    tempPos = // transform to GUI pos
    minBounds = Vector3.min(minBounds, tempPos);
    maxBounds = Vector3.max(maxBounds, tempPos);

// draw bounds
// profit!

Wattosan · September 8, 2017, 5:49am

I tried using this solution, however it seems the projection is done incorrectly for me. It draws the bounding box around the object correctly, but not the shadow:
https://drive.google.com/open?id=0B0gogWMq7zHXU2RYWFNWUlliS1U

The code now looks like this:

        float planeY = 0;

        Vector3 minBounds = new Vector3(Mathf.Infinity, Mathf.Infinity, Mathf.Infinity);
        Vector3 maxBounds = new Vector3(Mathf.NegativeInfinity, Mathf.NegativeInfinity, Mathf.NegativeInfinity);
        Vector3 tempPos = Vector3.zero;
        Vector3 cornerPoint = Vector3.zero;

        foreach (Transform pointLightTransform in pointLightTransforms)
        {
            for (int i=0; i < pts.Length; i++)
            {
                cornerPoint = pts[i];
                Vector3 pointLightDir = (cornerPoint - pointLightTransform.position).normalized;
                planeY = planeTransform.position.y;
                tempPos += -pointLightDir * ((cornerPoint.y - planeY) / pointLightDir.y); // Project to plane
                tempPos = cam.WorldToScreenPoint(cornerPoint); // Transform to screen pos
                tempPos.y = Screen.height - tempPos.y; // Transform to GUI pos
                minBounds = Vector3.Min(minBounds, tempPos);
                maxBounds = Vector3.Max(maxBounds, tempPos);
            }
        }

        Rect r = Rect.MinMaxRect(minBounds.x, minBounds.y, maxBounds.x, maxBounds.y);
        GUI.Box(r, "Shadow");

For some reason I feel as if this is an easy fix.

bgolus · September 8, 2017, 7:49am

Look carefully at how you’re using tempPos and cornerPoint…

(Hint: remove one entirely and replace it with the other )

Wattosan · September 8, 2017, 8:59am

But of course…My bad.

tempPos = cam.WorldToScreenPoint(cornerPoint); // Transform to screen pos

should be changed to

tempPos = cam.WorldToScreenPoint(tempPos); // Transform to screen pos

Now it works perfectly! Thank you! We are almost there…It works not so perfectly on objects with multiple meshrenderers. Let’s pretend I have a bed and it consists of multiple mesh renderers. For each mesh renderer I find the 8 points of the bounds. And then try to find min and max. It seems to be working kind of right. Except that there is a lot of free gap between the edges of the bounding box and the actual mesh.

Another video, which first demonstrates that the solution you’ve helped me achieve works with multiple point and directional lights (even combined together), on objects with a single mesh renderer. Then it demonstrates that things get messy as soon as an object consists of more than 1 mesh renderer:

https://www.youtube.com/watch?v=FtQqkY9WXc0

The changed code looks like this:
Code

[SerializeField] Camera cam;
    [SerializeField] Transform planeTransform;

    Vector3[] pts = new Vector3[8];
    List<Light> lights = new List<Light>();

    public GameObject _target;
   
   

    private void Start()
    {
        foreach (Light l in FindObjectsOfType<Light>())
        {
            if (l.type == LightType.Point || l.type == LightType.Directional)
            {
                lights.Add(l);
            }
        }
    }

    void OnGUI() {
        List<MeshRenderer> meshRenderers = _target.GetComponentsInChildren<MeshRenderer>().ToList();
        if (meshRenderers.Count >= 1) {
            pts = new Vector3[8 * meshRenderers.Count];
            int counter = 0;
            foreach (MeshRenderer mr in meshRenderers)
            {
                Bounds mrBounds = mr.bounds;

                if (cam.WorldToScreenPoint(mrBounds.center).z < 0) { Debug.Log("returning"); }

                pts[counter + 0] = new Vector3(mrBounds.center.x + mrBounds.extents.x, mrBounds.center.y + mrBounds.extents.y, mrBounds.center.z + mrBounds.extents.z);
                pts[counter + 1] = new Vector3(mrBounds.center.x + mrBounds.extents.x, mrBounds.center.y + mrBounds.extents.y, mrBounds.center.z - mrBounds.extents.z);
                pts[counter + 2] = new Vector3(mrBounds.center.x + mrBounds.extents.x, mrBounds.center.y - mrBounds.extents.y, mrBounds.center.z + mrBounds.extents.z);
                pts[counter + 3] = new Vector3(mrBounds.center.x + mrBounds.extents.x, mrBounds.center.y - mrBounds.extents.y, mrBounds.center.z - mrBounds.extents.z);
                pts[counter + 4] = new Vector3(mrBounds.center.x - mrBounds.extents.x, mrBounds.center.y + mrBounds.extents.y, mrBounds.center.z + mrBounds.extents.z);
                pts[counter + 5] = new Vector3(mrBounds.center.x - mrBounds.extents.x, mrBounds.center.y + mrBounds.extents.y, mrBounds.center.z - mrBounds.extents.z);
                pts[counter + 6] = new Vector3(mrBounds.center.x - mrBounds.extents.x, mrBounds.center.y - mrBounds.extents.y, mrBounds.center.z + mrBounds.extents.z);
                pts[counter + 7] = new Vector3(mrBounds.center.x - mrBounds.extents.x, mrBounds.center.y - mrBounds.extents.y, mrBounds.center.z - mrBounds.extents.z);
                counter += 8;
            }
        }

        // 3. Project corners onto plane
       
        float planeY = 0;

        Vector3 minBounds = new Vector3(Mathf.Infinity, Mathf.Infinity, Mathf.Infinity);
        Vector3 maxBounds = new Vector3(Mathf.NegativeInfinity, Mathf.NegativeInfinity, Mathf.NegativeInfinity);
        Vector3 tempPos = Vector3.zero;
        Vector3 cornerPoint = Vector3.zero;
        Vector3 pointLightDir = Vector3.zero;

        foreach (Light l in lights)
        {
            for (int i=0; i < pts.Length; i++)
            {
                cornerPoint = pts[i];
                pointLightDir = l.type == LightType.Point ? (cornerPoint - l.transform.position).normalized : l.transform.forward;
                planeY = planeTransform.position.y;
                tempPos = cornerPoint + (- pointLightDir * ((cornerPoint.y - planeY) / pointLightDir.y)); // Project to plane
                tempPos = cam.WorldToScreenPoint(tempPos); // Transform to screen pos
                tempPos.y = Screen.height - tempPos.y; // Transform to GUI pos
                minBounds = Vector3.Min(minBounds, tempPos);
                maxBounds = Vector3.Max(maxBounds, tempPos);
            }
        }

        // 6. Draw bounds
        Rect r = Rect.MinMaxRect(minBounds.x, minBounds.y, maxBounds.x, maxBounds.y);
        GUI.Box(r, "Shadow");

Any ideas how to remove the empty space?

bgolus · September 8, 2017, 5:26pm

That’s in part a product of using the renderer bounds rather than the mesh’s local bounds. Any object that’s slightly rotated is going to get a significantly larger renderer bounding box because the renderer bounds is world space axis aligned. Try placing a box that covers the entire object with out rotating it, that’s what you’re actually projecting the corners of. The mesh’s local bounds will be much smaller, though something like a bed will always have a smaller shadow than calculated. Again, you’re just projecting a box, so the foot of the bed is being treated just as tall as the headboard.

In some ways this is the same problem as doing physics collisions on complex objects. You want a simplified representation of the object that still matches the visual bounds of the object. For complex objects like this you may need to resort to manually placing a small number of proxy boxes that conform to the shape of the object you want to know the shadow bounds of. Then iterate over each proxy & light combination and find that min-max. If your mesh is multiple sub meshes they should have their own local bounds you could use.

However the other problem is the code makes no allowances for how visible a shadow is. In that first example from that video the bounds are “correct”, but the shadow from the second light isn’t visible either due to the relative brightness, or the light’s range, etc. To solve this you will basically have to calculate all of the lighting the GPU is doing on the CPU by hand, figuring out falloff and perceptual contribution of the shadows, etc.