Disable Ambient Light for some GameObjects

I have some GameObjects that are not affected by the settings under Lighting > Scene > Ambient Color while some are affected.

I would like to be able to control some of them to be affected or not affected by the Ambient Color. How do I go about doing this?

An objects ability to be affected by the lighting in a scene will all come down to the material that is used to render it.

You will notice if you set the material on an object to use an unlit shader, then it wont be affected. Switch it back to a non-unlit shader and it should start being affected again by lighting.