Is a primitive Cube removed off mesh and renderer the same as an Empty, or does it still have references for material or whatever 
The references are on the components, not the gameObject itself. If the components are gone, the references are gone.
Would this would mean a cube with no components is treated the exact same way as an Empty in Unity?
Or is still somehow different, because it originally was something else? (e.g. using up more memory or so)
If you pick a cube, you remove all its component, you get just an empty game object. This is identical to creating an empty game object.
As you create a cube, Unity also needs to create an empty game object and then adds the relevant components. You just don’t see it.
Ok thanks a lot, now I’m 100% clear 