Does object size affect performance?

Will making objects really small and moving the camera closer make the game run faster?

And if so are there any drawbacks?

Game performance is based on the number of vertices each object has so given that they have the same number of vertices and you just change the scale then the performance should be the same.

As for the camera, performance is related to the number of visible objects at a time and not on the scale of objects. ie seeing 5 objects out of 10 will yield a better performance from a far away camera that shows everything at once.