How would one implement the equivalent to Transform.LookAt, but w/ angle constraints?

I’m attempting to implement floating UI elements that act like target brackets over targets. It’s a bit different than the typical situation where you can simply convert world units to screen coordinates, as I need the UI elements out in front of the player (currently 4 units out). It’s to support stereoscopic 3D, including Oculus Rift. It’s not as simple as just moving the elements out into the world, since they won’t line up properly. The UI elements need to live on a point on the line between the camera’s position and the target’s position.

To handle the basic lining up of the floating UI elements, I can do this and it works perfectly:

            // 1) Move UI element to the camera position
            _structureGUITransform.position = gameWorldCamera.transform.position;

            // 2) Face UI element toward target
            _structureGUITransform.LookAt(_cachedTransform.position);   // This works, but doesn't handle off-screen indicators

            // 3) Move 4.0 units along the UI element's forward vector
            _structureGUITransform.Translate(Vector3.forward * 4.0f);

            // 4) Rotate to match camera orientation
            _structureGUITransform.rotation = gameWorldCamera.transform.rotation;

So far, so good. But now I want to support icons that stop within a certain angle of the camera’s forward vector to act essentially as off-screen targets.

My thought was to replace #2 above with the equivalent of Transform.LookAt, but with the ability to constrain/clamp the X and Y rotation angles to whatever values I want relative to another vector, in this case the camera’s forward vector.

So before I worry about clamping the values, I need to get my version of Transform.LookAt working and I’m not there yet.

I replaced #2 with the below code and it almost worked:

            Vector3 cross = Vector3.Cross(direction, gameWorldCamera.transform.forward);
            var angleX = Mathf.Asin(cross.x) * Mathf.Rad2Deg;
            var angleY = Mathf.Asin(cross.y) * Mathf.Rad2Deg;

            _structureGUITransform.rotation = gameWorldCamera.transform.rotation; // Start aligned with camera
            _structureGUITransform.Rotate(angleX, angleY, 0);

The Y axis rotation (left and right) appears to work correctly. But the X axis (up and down) is off by roughly 50%. It does rotate in the proper direction, just not nearly enough. I’ve tried some “magic number” stuff just for grins, like multiplying the X angle by 1.6, but that only sort of works and doesn’t really make sense anyway.

Any ideas what I’m doing wrong here? It’s possible that my angleX calculation is off and that’s the only issue, but I’m not sure where to go from here.

After some more experimentation, I found that the X angle value is off by exactly 1 / 0.64, which is 1.5625. Why? No idea.

I think the math is wrong here anyway. Getting the individual X and Y angles based on only the direction vector to the target and the camera’s forward vector may not be enough. It seems like the up (and/or right) vector will need to be incorporated somehow. For instance, the ship can “roll”, which wouldn’t change either vector, but could change the position of the UI overlay.

Any thoughts on how I can accomplish this? The short version of the question is: how can I implement a Transform.LookAt type of thing, but with the ability to restrict the relative angle change?

Hehe, well apparently this isn’t a popular topic. But for sake of completion, I’ll post my solution.

I ended up breaking the logic into two pieces. When the target is within the “view cone”, I do my simple “point toward target” logic that was already working. Though I did change it from Transform.LookAt to Quaternion.LookRotation(direction), since I can do Quaternion.Angle to check the angle and go to the “in cone” or “out of cone” logic.

For the off-screen indicators, I projected the normalized direction vector onto the X/Y plane in front of the player by using the camera’s forward vector as the normal. Then I created a Ray based on that flattened/projected vector with the camera position as the origin. Then I did a Ray.GetPoint, passing in a hard-coded value to get a point along that ray. So instead of trying to rotate the UI element, I moved it to that point. After that I do my same trick of moving it along the UI’s forward vector to move it in front of the player.

There’s a little bit of “magic number” stuff going on with the GetPoint(3.0f) below, but I can play with it to get the transition between in the cone and out of the cone working smoother, but already it’s working well. I could even just use the GetPoint method for everything, but I do like the “spot on” accuracy of turning directly toward the target when in the cone.

Here’s the “out of cone” logic:

                Vector3 planeNormal = gameWorldCameraTransform.forward;
                Vector3 worldNormal = direction;

                Vector3 projectionOnNormal = Vector3.Project(worldNormal, planeNormal);
                Vector3 flattenedDirection = worldNormal - projectionOnNormal;

                Ray ray = new Ray(gameWorldCameraTransform.position, flattenedDirection);
                Vector3 newPos = ray.GetPoint(3.0f);    // Get a point 3.0 units along the ray

                _structureGUITransform.position = newPos;

                // Rotate to match camera (to align with the forward vector)
                _structureGUITransform.rotation = gameWorldCameraTransform.rotation;

                // 3) Move 4.0 units along the UI element's forward vector
                _structureGUITransform.Translate(Vector3.forward * 4.0f);

Maybe its actually very interesting… But a hard topic to answer. I tend to stick to topics that I know I can provide some sort of insight to. This is just more of an Advanced topic.