Physic Material does not affecting rolling sphere

I am trying to create a game where the player has to control a rolling sphere moving through a level.
The level consist of different types of terrain with different bounciness and friction, these are set with the help of a physic material.
My problem is that because the ball is rolling the friction of the terrain has no effect on the movement of the ball, even with a friction of 5000 the ball does not stop moving when it is rolling over the terrain.

So is there a better way to make the friction of a rolling ball be affected by the terrain it is rolling over?

Objects Rolling Forever

Let’s say you’re developing a golf game. There is a problem with how the golf ball does not stop rolling, and somehow manages to continue rolling on forever as long as there isn’t any kind of hole or ditch in its path. The reason why this happens is because in real life, the ball would be slowed down by the grass it runs over (among other things), since it has to push the tiny grass blades down, the grass essentially is like a constant ramp. This is called rolling resistance. Unity cannot simulate this behaviour accurately, so instead artificial stopping forces must be used.

In Unity the best force to use to stop an object from rolling forever is “angular drag”. Changing the angular drag on the golf ball is the way to fix this issue. The exact value really depends on the behaviour you’re looking for, however you may notice that a value of 1.0 angular drag may not even be enough in some cases.

Per this article, put an OnCollision on your object and when it hits different types of terrains, change the object’s angular drag at runtime.