Sometimes it happens when I pull, but it always happens when I clone the repo. Is there a specific element of the metadata that manages this connection that I need to be sure is being committed?
Go to Project Settings > Editor > Use Meta Files. This will generate .meta files for everything in your project, which will keep the references when you move the files around.