Avoiding collision "spaz" effect

I’ve got this gameObject which connects by script by a Character Joint to another gameObject controlled by the user by user input (touchscreen, buttons, movement controls, and so), and what happens is that when I move around and collide this attached gameObject I move around the place with me, this spazzes and dances around depending on the collision I make. It’s mostly when pushing this into the scenery for example.
What’s wrong here? How can I avoid this?

I assume the gameObject is a rigidbody, if u want the gameObject to collide with a specific collider use something like this

function OnCollisionEnter(col : Collision){
if(col.gameObject.CompareTag("the object you want your collider to collide with")){
Physics.IgnoreCollision(yourgameObject.collider, collider)
}
}