Help with raycast stopping player from going through walls

Hello I am using raycasts to stop the player from going through walls. I have a camera controller that allows the player to move around the 3d space similar to the way you can move around in the scene of a unity project. The method I use to stop the player from going through walls if they move forwards works perfect every time. I tried to use this same method when the player pans left or right. It works some of the time but I can still break through walls alot of the time. I can’t figure out why this isn’t working like my other raycast method any help would be appreciated. To activate that movement to pan left or right I have it set up so the user just needs to hold the middle scroll button down. Here is the raycast code for the left and right pan.

         //if the middle button is pressed down
        if (horizontalTranslation.isActivated())
        {
            //this code is what moves the camera
            float translateX = Input.GetAxis(mouseHorizontalAxisName) * horizontalTranslation.sensitivity;
            transform.Translate(translateX, 0, 0);
            //variables for the raycast
            origin = transform.position;
            Vector3 eLeft = GetComponent<Camera>().transform.right;
            Vector3 eRight = GetComponent<Camera>().transform.right;
            RaycastHit hitLeft;
            RaycastHit hitRight;
        //raycast that checks left side if there is a hit close enough it loads the last saved 
        //position instead of moving in the direction the user wants to move
        if (Physics.SphereCast(origin, sphereRadius, eLeft, out hitLeft, maxDistance, layerMask, QueryTriggerInteraction.UseGlobal))
        {
            //checks if raycast hit is in between 1 and 5
            Debug.LogWarning("left initial hit " + hitLeft.distance);
            if ((hitLeft.distance > 1) && (hitLeft.distance < 5))
            {
                originalRayPos = transform.position;
                originalRayRot = transform.rotation;
                Debug.LogWarning("left set " + hitLeft.distance);
            }
            //if raycast hit is less than 1 the last saved position is used to stop player from going that direction
            if (hitLeft.distance < 1)
            {
                transform.position = originalRayPos;
                transform.rotation = originalRayRot;
                Debug.LogWarning("left use " + hitLeft.distance);
            }
        }
        //raycast to check the right side if there is a hit close enough it loads the last saved 
        //position instead of moving in the direction the user wants to move
        if (Physics.SphereCast(origin, sphereRadius, eRight, out hitRight, maxDistance, layerMask, QueryTriggerInteraction.UseGlobal))
        {
            Debug.LogWarning("left initial hit " + hitRight.distance);
            if ((hitRight.distance > 1) && (hitRight.distance < 5))
            {
                originalRayPos = transform.position;
                originalRayRot = transform.rotation;
                Debug.LogWarning("left set " + hitRight.distance);
            }
            if (hitRight.distance < 1)
            {
                transform.position = originalRayPos;
                transform.rotation = originalRayRot;
                Debug.LogWarning("left use " + hitRight.distance);
            }
        }

If anyone else is struggling with this issue I found a better solution heres my code.

    if (horizontalTranslation.isActivated())
    {
        //sets translate to be used for transformation
        float translateX = Input.GetAxis(mouseHorizontalAxisName) * horizontalTranslation.sensitivity;
        //variables for raycasting
        origin = transform.position;
        Vector3 eLeft = -GetComponent<Camera>().transform.right;
        Vector3 eRight = GetComponent<Camera>().transform.right;
        RaycastHit hitLeft;
        RaycastHit hitRight;
        //raycast sent to right
        if (translateX > 0)
        {
            if (Physics.SphereCast(origin, sphereRadius, eRight, out hitRight, maxDistance, layerMask, QueryTriggerInteraction.UseGlobal))
            {
                //do nothing if hit is detected
            }
            else
            {
                //transform position if no hit detected
                transform.Translate(translateX, 0, 0);
            }
        }
        //raycast sent to left
        if (translateX < 0)
        {
            if (Physics.SphereCast(origin, sphereRadius, eLeft, out hitLeft, maxDistance, layerMask, QueryTriggerInteraction.UseGlobal))
            {
                //transform.Translate(translateX, 0, 0);
            }
            else
            {
                transform.Translate(translateX, 0, 0);
            }
        }    
    }