I'm just wondering if the Scale of an object makes a difference. For example if all of my objects were huge and the camera was simply zoomed further out would that take any more processor or memory than the same objects but smaller and the camera zoomed closer?
Short answer: no, but you don't want to do that
Overall performance is not influenced by the general scale of your scene - you still need to keep the same objects in memory, you still need to render all the polygons, etc.
However, Unity uses the "float" type for all math and transformation operations, which has very limited precision (but uses little memory). So if you scale your whole scene by a factor of say 100000 (and adjust the clipping planes of the camera accordingly), you can no longer work with models that are scaled around 1 because of limited floating point precision (for example, Unity cannot distinguish between the positions (1000000,0,0) and (1000001,0,0).
Especially the Physics engine and anything related to it relies on "reasonable" values, ideally around 1, and you often get unpredictable results outside the interval of say 0.01-1000 (depending on the parameter).
This is also the reason why a scale of "meters" is generally a good idea - but only as long as you use "normal" objects: If you want to visualize the Solar System, you don't want to use meters, you'd probably use Astronomic Units or Millions of km. And if you want to visualize atoms, you don't model them in meters, but maybe in nanometers.
In theory it shouldn't matter much, aside perhaps from rotational inertia tensors, I'm not sure.
However, in practice, it becomes really hard to create realistically looking worlds if you don't stick to a certain set of units.
I prefer to always think in meters, seconds and kilograms. This way I can create objects whose size, mass or velocity/etc I know roughly, and I know that they will fit into my world because they are based on the same units as the other objects. Otherwise you will spend lots of time tweaking scaling on objects all the time.. which is annoying. If you stick to a set of units, you get it right the first time, mostly.