Its easy to remove Specular form lamps (Just uncheck affect specular.
But how to stop HDRI SKY from affecting Specular?
Example of HDRI SKY Specular reflection on pine tree.
Even by setting Specular collor to Black and Smoothness to 0 we still have this Specular reflection (top picture)

Directional lights have this Checkbox.
Is there way to block Specular reflections alltogether?