How do I change the manipulator/pointer (I dont know what its actually called) to show at the exact point of the game object, ignoring the children in the scene view.

I assume how it works now is the pointer is at the average position of itself and its children. How do I get it to ignore the children?

I thounk you mean the transform gizmo. Above the scene view there are two buttons concerning the pivot and the shown transform orientation. Change the pivot from global to local and it should show at the position of the currently selected gameobject.