Here is the code that I am using to rotate my object towards the mouse cursor:
public float curSpeed;
public float maxSpeed = 40;
public float rotationSpeed = 3.5f;
public LayerMask targetLayer;
Ray ray;
RaycastHit hit;
Vector3 targetPos;
Quaternion quaternion;
void FixedUpdate()
{
ray = Camera.main.ScreenPointToRay(Input.mousePosition);
if (Physics.Raycast(ray, out hit, 200, targetLayer))
{
targetPos = hit.point;
curSpeed = (Vector3.Distance(transform.position, targetPos) * (maxSpeed / 6)) - (maxSpeed);
if (curSpeed > maxSpeed)
curSpeed = maxSpeed;
else if (curSpeed < 0)
curSpeed = 0;
quaternion = transform.rotation;
transform.LookAt(targetPos);
transform.rotation = Quaternion.RotateTowards(quaternion, transform.rotation, rotationSpeed);
}
rigidbody.AddRelativeForce(0, 0, curSpeed);
}
This works perfectly (I have a box collider set up with a Y value of 0 at (0, 0, 0), this is what the raycast collides with) until my object (a ship) collides with another ship (I use rigidbodies and mesh colliders (the other ship is convex, so collision does work)) at which point my player controlled ship starts rotating oddly: it rotates significantly faster in one direction than the other. This seems to go away after I move it around a bit, but it takes a bit.
Ideas on what is going on? Is this a problem with the rigidbody collision? (It is what causes the problem, but is it inherent in the collision system? Do I need to make my own system in order to fix this?)