Friction not applied in fast collisions using Continuous collision detection modes

Starting in Unity 2022.3 versions, this started happening -
In my scenario, I have a ball bouncing on a surface (e.g on concrete) and no matter what the friction values are, the ball basically ignores ang velocity/friction impact. So it bounces and the friction/ang velocity are basically not applied.

When changing the collision detection mode to Discrete, it works perfectly fine.
When changing it to one of the continuous modes, it breaks.
I am using Sphere Collider for the ball of course.

I went through each physic settings - nothing helped.