Making Rigidbodies to interact but allow them to go one through the other

In my game I have a Player and many leaves shaken by the wind.

I have activated Colliders and Rigidbodies in both elements so they can interact with other elements.

I am trying the Player to interact with the leaves dragging them a bit but allowing the Player to go through them with minimum resistance.

I know I can play with the Layer Collision Matrix to make the Leaves ignore the Player but I actually want some kind of interaction as explained above.

This is what I have now with the Layer Collision Matrix and the Colliders and the Rigidbodies all activated:

Any suggestion of how can I achieve this? Is there something I can do with the Unity physics or should I have implement this procedurally with code?

Make the collision a trigger then you can use OnTrigger() for your interaction and still be able to pass through them.