How to limit where my SphereCast can hit?

I’ve been working on a solution for this for over a week and at this point I’m out of ideas.
slope issue problem

My SphereCast (or even Raycast) should only be able to work on the object to the right (50 degrees slope object) where my player will slide down to the ground. However, it is also performing the same sliding action on the object to the left and other smaller sized objects like it which is causing very glitchy/jittery movement and prevents my player from simply walking over a small object like that.

void FixedUpdate()
{
        RaycastHit hit;

        if (Physics.SphereCast(transform.position, radius, Vector3.down, out hit, 2f))
        {
            hitPointNormal = hit.normal;
            float slopeAngle = Vector3.Angle(hitPointNormal, Vector3.up);
            
            if (slopeAngle > controller.slopeLimit)
            {
                isSliding = true;
            }
            else
            {
                isSliding = false;
            }
        }
}

The only method I know is to use a Layer Mask, but what if I have many small objects in both the same layer and different layers?

If you want to slide on some slopes, but not others, you kinda have to mark the slopes that you’re supposed to slide on in some way. Either with layers, or with a marker script, or with a tag. I’d go with a marker script as that’s the most flexible, as it doesn’t interfere with other uses of layers or tags.

I suppose you could do a thing where you do a bunch of raycasts instead of just one, and if several of them hit flat ground instead of a slope, then you’re not considered on a slope. That does seem harder to get right than just “the ground has the slippery slope script on it”