rigidbody.centerOfMass not relative to object rotation

Ok I am attempting to run Physics.Raycast based on the center of mass of my object. This works fine at first, but when My object rotates it the center of mass is not recalculated.

For example, I have a rectangular prism that is 2,1,1. It is a custom mesh imported from blender and its origin is intentionally offset so that when I rotate it, it can rotate around a pivot point that is snapped to a grid rather than rotating around its apparent center of mass. This is the desired behavior, however I need to run the Raycast from its center of mass. Still not a problem, rigidbody.centerOfMass returns 0,0,-0.5 and I simply add my objects transform to its center of mass to get the location that it should cast the ray from. So far so good.

Here is the problem. When I rotate the object, it does not update the center of mass to reflect the change. So that it still returns 0,0,-0.5 regardless of my objects rotation.

How would I apply my objects rotation to the rigidbody.centerOfMass so that I can add it to the objects transform and get its actual center of mass?

Thanks in advance for the help, I know this is kind of a weird question.

A chunk of relevant code:

//Trace Down
	Debug.Log(rigidbody.centerOfMass);
	ray = Ray(transform.position + rigidbody.centerOfMass, -transform.up);
	distance = (GetComponent(BoxCollider).size.y / 2) + 0.1;
	Debug.DrawLine(ray.origin, ray.GetPoint(distance), Color.red, 5, false);
	MyHit = Trace(ray.origin, ray.direction, distance);
	if(MyHit != null)
	{
		Faces[0] = MyHit.HitObject;
	}
	else
	{
		Faces[0] = null;
	}

Center of mass is a local coordinate. You can use Transform.TransforPoint() to find the world position:

var pos = transform.TransformPoint(rigidbody.centerOfMass);
ray = Ray(pos, -transform.up);