Turret rotation with a pivot?

I have a unit with a gun gameobject attached to its body. I run into a problem when I tell the gun to lookup at the gray square on the black gameobject (See Picture).

The gun looks up at the gameobject but it also moves itself off of the unit gameobject. I want the side of the gun that is connected to the gameobject to be anchored to the body. I’ve tried using joints to fix this problem but still not sure if this would be the best solution. Any help would be appreciated!

Code used to make gun turret look at gray “LookAt” gameobject:

Quaternion LookObjRot = Quaternion.LookRotation (LookAt.transform.position - controller.Gun.transform.position);
controller.Gun.transform.rotation = Quaternion.Slerp (controller.Gun.transform.rotation, LookObjRot, Time.deltaTime * 5);

The easiest way to do this is to create an empty parent object for the turret to act as its pivot point.

Then, rather than rotating the turret itself, you rotate that empty GameObject:

controller.GunPivot.transform.rotation = etc.