Character controller, player y changes when pushing on colliders

So I am trying to use a character controller. This is a 3rd person, not first person, controller if it makes a difference. Now everything is fine until I hit a collider. I have several trees in my scene and if I continue to walk into them or walk into specific spots of my trees my character basically gets pushed up the tree’s collider rather than staying flat on the ground. The y also changes when I push against my invisible wall colliders around the bounds of the plane in my scene. My invisible wall colliders are box colliders, all other colliders are meshes.
Below is my current code, this function gets called in Update:

 private void Movement()
    {

        //Player object movement
        float horMovement = Input.GetAxisRaw("Horizontal");
        float vertMovement = Input.GetAxisRaw("Vertical");


        if (horMovement != 0 && vertMovement != 0)
        {
            speed = defaultSpeed - .5f;
                       

        }
        else
        {
            speed = defaultSpeed;

        }
        Vector3 moveDirection = new Vector3(horMovement, 0,vertMovement);
        moveDirection = transform.TransformDirection(moveDirection);
        moveDirection *= speed;
        if ((moveDirection * Time.deltaTime).x !=0f && (moveDirection * Time.deltaTime).z != 0f)
        {
            Debug.Log(moveDirection * Time.deltaTime);
        }
        Vector3 fixedTransform=new Vector3(moveDirection.x,0,moveDirection.z);
        GetComponent<CharacterController>().Move(fixedTransform * Time.deltaTime);
       
        //Player graphic rotation
        Vector3 rotateDirection = new Vector3(horMovement, 0, vertMovement);
        if (moveDirection != Vector3.zero)
        {
            Quaternion newRotation = Quaternion.LookRotation(rotateDirection);
            playerGraphic.transform.rotation = Quaternion.Slerp(playerGraphic.transform.rotation, newRotation, Time.deltaTime * 8);

        }
    }

I would like for the y to never change at all. Any and all help is appreciated.

oncollision you can freeze the player’s Y position.
Something like this maybe?

private float currentY;
void OnCollisionEnter(Collision coll) {
currentY = gameObject.transform.position.y;
}

void OnCollisionStay(Collision coll) 
{
transform.position = new Vector3(transform.position.x, currentY, transform.position.z);
}