Why does changing the velocity on my object make it only move in one direction, but not the other?

I am making a endless runner kinda game (think Subway Surfers) with similar swipe mechanics that I was in the process of making. I coded in the swipe controls in the PlayerController script as you can see.
Problem is that when I swipe right it works just fine, but swiping left does not move the player. The debug.log that I added shows that there is a velocity being applied but the player is still not moving for some reason. If someone could help it would be really appreciated.

private Vector2 startTouchPosition;
private Vector2 endTouchPosition;
private int position = 1;
private Rigidbody rb;
public float xSpeed;
private bool swipeLeft = false;
private bool swipeRight = false;
// Start is called before the first frame update
void Start()
{
    rb = GetComponent<Rigidbody>();
}

// Update is called once per frame
void FixedUpdate()
{
    if(Input.touchCount > 0 && Input.GetTouch(0).phase == TouchPhase.Began)
    {
        startTouchPosition = Input.GetTouch(0).position;
    }

    if(Input.touchCount > 0 && Input.GetTouch(0).phase == TouchPhase.Ended)
    {
        endTouchPosition = Input.GetTouch(0).position;

        if(endTouchPosition.x < startTouchPosition.x)
        {
            if(position > 0)
            {
                swipeLeft = true;
                position--;
                Debug.Log("Left");
            }
                     
        }

        if(endTouchPosition.x > startTouchPosition.x)
        {
            if (position < 2)
            {
                swipeRight = true;
                position++;
                Debug.Log("Right");
            }
        }
    }

    if(swipeLeft == true && transform.position.x >= -3)
    {
        rb.velocity = new Vector3(-1000, rb.velocity.y, rb.velocity.z);
        rb.useGravity = false;
        Debug.Log("Swiped left" + rb.velocity);
    }
    else
    {
        rb.velocity = new Vector3(0, rb.velocity.y, rb.velocity.z);
        rb.useGravity = true;
        swipeLeft = false;
    }

    if(swipeRight == true && transform.position.x <= 3)
    {
        rb.velocity = new Vector3(xSpeed, rb.velocity.y, rb.velocity.z);
        rb.useGravity = false;
        Debug.Log("swiped right" + rb.velocity);
    }
    else
    {
        rb.velocity = new Vector3(0, rb.velocity.y, rb.velocity.z);
        rb.useGravity = true;
        swipeRight = false;
    }
}

}

Hello.

First i see is a problem in logic of your code, (i think, i dont know eactly how your project is and what you want to acheive) look:

if (A & B) make W
else make X
if (C & D) make Y
else make Z

you did this structure in your code. the problem is that code is executing each fixed update W or X plus Y or Z. I mean, it will be executing 2 results each time, for example W & Y or X & Y or W & Z

Understand what i mean?
You dont have 4 different situations where the code only executes one. You have 2 situations 2 times

Maybe you want something like (only one result will be true):

if (A & B) make W
else if (A & !B) make X
else if (C & D) make Y
else if (C & !D) make Z

or something like this… (think on your project, whats the exact thing you want.

Good luck!