Affect only Specular

Can I have point lights affect only specular and not diffuse (color)?

Here is what I want to do.

I have an interior, lightmapped, and it’s lit with a sun, so it has indirect light mostly.

I want to put some point lights at places where there is a lot of light to fake some subtle “specular reflections”. But the diffuse shading is already “correct” with the lightmapping, so I don’t want more light affecting that.

I’m assuming there is no easy way of doing this, right?

Do I have to use custom Shaders? How do I make a shader ignore certain lights for only certain types of shading…?

Sounds like a simple Shader change to ignore diffuse lighting. Maybe ask in the ShaderLab section.
The builtin shaders are available for download somewhere, so you can play around with them.

Thing is I don’t want to ignore diffuse lighting from all lights… Just from certain lights. I don’t think I can do that with a shader (right?).