Preserve horizontal speed on slopes

Hey guys
I have a 3D character controller with rigidbody that moves along a grid. Basically, if you swipe right, the character will move right until he hits a wall.
I also have gravity so my player can jump.

The issue is, when my player climbs a slope or stairs, he slows down drastically, because gravity is pulling him down. Is there a way to preserve my horizontal speed even on upward slopes?

Here is my movement code

//If we are not moving...
            if (!isMoving)
            {
                //If requested direction is valid, set it to currentDir and rotate character
                if (CheckDirection(nextDirection))
                {
                    currentDirection = nextDirection;
                    RotateCharacter(currentDirection);
                }
                //If we are stopped, calculate the next valid start and end position
                startPosition = rb.position;
                endPosition = startPosition + (currentDirection * unitLength);
                //Round the result so we end up on a whole number coordinate
                endPosition = new Vector3(Mathf.Round(endPosition.x), Mathf.Round(endPosition.y), Mathf.Round(endPosition.z));
                isMoving = true;
            }

            //If we are moving and the current direction is valid (no walls in the way)
            if (isMoving && CheckDirection(currentDirection))
            {
                //Move rigidbody towards destination
                float step = speed * Time.deltaTime;
                rb.MovePosition(Vector3.MoveTowards(rb.position, new Vector3(endPosition.x, rb.position.y, endPosition.z), step));
            }
            else
            {
                RotateCharacter(nextDirection);
            }

            //Calculate horizontal distance remaining from startpoint to endpoint
            Vector3 distanceRemaining = new Vector3(endPosition.x, 0, endPosition.z) - new Vector3(rb.position.x, 0, rb.position.z);

            //If we are extremely close to endpoint, stop
            if (distanceRemaining.sqrMagnitude < float.Epsilon)
            {
                isMoving = false;
            }
            //If we hit a wall, stop
            if (!CheckDirection(currentDirection))
            {
                isMoving = false;
            }

A few things which come to mind are do away with Unity’s physics and use a raycast system for collisions which applies gravity in the players transform direction. Change the vector of gravity to your players transform.down direction using Physics.gravity which would mean you player may walk on walls though or modify the force when traversing a slope.