rigidBody collision with wall stops vertical movement.

I am using the code below to move a group of GameObjects.
The script is on an empty GameObject which also has a RigidBody component. One of the child GameObjects has a BoxCollider component.

The problem

When the gameObject is above ground (in free fall) and collides with a wall, the gameObject gets stuck if the user is moving towards the wall making the object “cling” to the wall at the collision point. I want it to continue falling. How do I stop it from halting on the side of a wall?

The code

void FixedUpdate()
    {
        velocity = rigidbody.velocity;
        velocity.x = Input.GetAxisRaw("Horizontal") * moveSpeed;
        velocity.z = Input.GetAxisRaw("Vertical") * moveSpeed;
        rigidbody.velocity = velocity;
    }

that seems wierd, are you looking for an in depth complete answer or a quick hack?
Cause the quick hack is on collision if falling knock there ass back a quarter inch.