Drifting Raycast in Camera?

I'm making basic item pickup like you would see in most first person games (Oblivion, Bioshock). Point the center of the camera to the item, click, and you have the item in your inventory. So I have a script attached to a camera (parented to an out of the box character controller) that sends out a raycast and then do some logic on the game object it hits.

hit = new Raycast()
void Update(){
if (Physics.Raycast(transform.position, transform.forward, out hit, 75f)){
    //we hit something
    lastOrigin = transform.position;
    if(hit.transform != null){          
    lastPoint = hit.transform.position;
//if we want to interact with an object
if (Input.GetMouseButtonDown(0) || Input.GetButtonDown("Fire1"))  //Left Click
  //check if we're looking at any object
  if ((hit.collider != null) && (hit.collider.gameObject != null) )
//Draw where the last valid raycast was.
Debug.DrawLine(transform.position, lastPoint, Color.red);

Now, this works. To a point. I noticed after awhile my collisions were sort of flickering. So I ran a debug draw to see the raycast.

And got this. http://imgur.com/a/ukOaZ

If my screencaps aren't enough to explain it I noticed the origin point of the raycast was slowly sinking into the ground, but popping back to the place I wanted it to be. That means the transform position of the camera is drifting?! What?! Is raycast affected by Gravity by default? Is my Camera? Am I drawing the right thing? Has anyone else found this? Can anyone explain it?

Figured it out. Looks like cameras hate being parented to gameobjects with rigidbodies. They must be inheriting some unseen property that's causing them to be affected by gravity, or at least the physics components that I'm declaring inside them are affected at any rate.

Removed the rigidbody from my parent object and the drifting stopped.

I would like to bump this. I have a ray cast checking the normal of the ground and it works until the ray cast in below the terrain! this makes if (Physics.Raycast(ray, out hit)) return false intermittently until its far below. if I jump it starts working again. Iv changed ray.origin and the Ray instantiation with transform.position even debugged “printing” the transform.position is as expected… the ray.origin prints as expected pos but Debug.DrawLine(transform.position, hit.point, Color.red);
drifts down! sometimes the direction will flip to look at the hit point without if (Physics.Raycast(ray, out hit)) is removed…
ill look into the rigid body effects soon. anyone know of any documentation?