History:
My project finally exceeded a file size limit on BitBucket. My project is just over 16GB.
I tried to revert via Sourcetree, ended up screwing up my repo somehow.
And tbh I’ve never been stoked on my repo setup between Sourcetree and BitBucket, so I just want to start over.
Current Thinking:
After reviewing pros and cons of Git/ Azure Devops/ Gitlab, etc.
It’s looking like Azure is the best option for large projects due to lack of repo size limitations.
Questions:
I have several large files, and I’m concerned about the initial file upload limits. Is there a way around this? For example I believe Azure’s is 5gb.
I’m also concerned about the file size limits, as I have a few files in the 100s of MB. Again, I believe Azure’s is 25mg. Best way to deal with this?
Would it be the craziest thing to just ignore repo altogether, and use something like Google Drive for a backed up folder on the cloud? I’m a solo dev, and will be until the end of the project. So no need for branches. Am I nuts to consider it?
Once you get to the size you speak of, you need to pay for some kind of hosting that will allow that.
I don’t keep up on the offerings; I use bitbucket, GitHub and gitlab to mirror most of my repos.
I also never commit large files like any of the EXRs built for lighting, etc.
Alternately you can just push the repo to another hard drive such as a USB-attached drive.
Or if you really feel like it, set up a NAS and host your own git server, but that seems like overkill.
Here’s my standard blurb:
PROPERLY CONFIGURING AND USING ENTERPRISE SOURCE CONTROL
I’m sorry you’ve had this issue. Please consider using proper industrial-grade enterprise-qualified source control in order to guard and protect your hard-earned work.
Personally I use git (completely outside of Unity) because it is free and there are tons of tutorials out there to help you set it up as well as free places to host your repo (BitBucket, Github, Gitlab, etc.).
You can also push git repositories to other drives: thumb drives, USB drives, network drives, etc., effectively putting a complete copy of the repository there.
As far as configuring Unity to play nice with git, keep this in mind:
I usually make a separate repository for each game, but I have some repositories with a bunch of smaller test games.
Here is how I use git in one of my games, Jetpack Kurt:
Using fine-grained source control as you work to refine your engineering:
Share/Sharing source code between projects:
Setting up an appropriate .gitignore file for Unity3D:
Generally the ONLY folders you should ever source control are:
Assets/
ProjectSettings/
Packages/
NEVER source control Library/ or Temp/ or Logs/
NEVER source control anything from Visual Studio (.vs, .csproj, none of that noise)
Setting git up with Unity (includes above .gitignore concepts):
It is only simple economics that you must expend as much effort into backing it up as you feel the work is worth in the first place. Digital storage is so unbelievably cheap today that you can buy gigabytes of flash drive storage for about the price of a cup of coffee. It’s simply ridiculous not to back up.
If you plan on joining the software industry, you will be required and expected to know how to use source control.
Source control does require learning, but there are TONS of tutorials and courses and online reference.
You should strive to use source control as confidently as you use your file/folder system.
“Use source control or you will be really sad sooner or later.” - StarManta on the Unity3D forum boards
Start with self-hosted Git for a little while, and have a “local” working repository and a pretend “cloud” master repository to push changes to, so you can get familiar with the workflow.
If a chunk of your large binary assets are somewhat centralized and have few changes, consider making that piece a second repository. Then you can just link that folder to the first as an official dependency, or simply have an informal dependency by ignoring that folder and rely on snapshot backups for it instead. If you go to the cloud later, you’ll already be familiar with how you’re dealing with this blob group and might save time, space, money keeping it off the cloud.
one thing to ask, you are sure you’re excluding all the things you need to? git does have a large file option so git can handle it but you probably do want to look in hosting your own
I havent tried it but x10hosting offer git hosting as part of their web system
You can easily corrupt your project beyond repair by using cloud sync merely by using it yourself in two locations (devices). It will also sync the entire project, including the Library folder which is easily by far the biggest folder in your project both in terms of occupied disk space and number of files.
Because it hasn’t been mentioned: Unity Version Control (formerly Plastic) is also considered pretty good.
Reporting back as I found https://rclone.org/
There is some information on version controlling. I haven’t set this up yet, need to do a lot more reading. But if I figure this out as an option I’ll drop the info here.
(I see nothing on that page that describes Version Control. Just synchronized cloning.)
Cloning a file structure isn’t the same as Version Control. It’s basically the same as Dropbox or OneDrive. It’s copying data to have it in more than one place. This has issues, especially with a file-granularity scheme like this. If you just let it decide to sync, it lacks a “single source of truth,” where you specifically say THIS is the official version and THAT is the backup or working copy. You accidentally delete or change one thing, and the sync system just assumes that was intentional and makes an identical change to your other copy. If you have to manually sync in either direction, great, you have just doubled the total storage and used a bit of bandwidth to do it, but you still can’t go back more than one change. As CodeSmile says, it’s also dangerous when you have two working copies and one shared cloud version.
Version Control keeps an overall change history. MacOS has long had a built-in no-frills change history system called TimeMachine, where you can say “hey, I prefer what I had three changes back on this file, restore it.” With a more complete VC system, you can say “hey, I like this snippet of code from three changes back” or “I want all of the files in this group to go back to Tuesday whether that was 3 or 2 or 0 changes ago.” Or “mark the whole system as ‘Version 3’ so I don’t have to count how many changes or what calendar date it was, later.” It works well with multiple copies, and can tell you which person or which laptop was responsible for a change that got pushed to the master repository, or allow you to merge and incorporate changes all at once or in parts. It can keep track of “this file belongs in the demo branch, but not in the final release branch.”
Yup, my journey led me back to version control ultimately.
So I’ve figured out where I’m stuck - my .git/objects folder is massive, approx 10gb. It’s the literal versions taking up 60+% of the space.
I’ve been searching for a safe way to clean this up, even back it up somewhere else. But without finding info on this, and not really knowing what these files are or do, I’m hesitant to update/ move/ delete these files/ folders.
In an ideal world I can just copy paste these files somewhere else, and start a fresh respository with no previous versions. Then if I ever needed those versions, I have them on another hard drive or something. Ultimately it’s all for safety, and I really don’t need any of the versions pre-dating last year (same project since 2019).
Basically, archive your whole project repo with a snapshot to an external hard drive you can put in your mother’s closet for a decade.
Then you’ll want to use git replace, or the older but still valid git rebase approach, to make a fresh repo with LIMITED history as of a known good recent commit. This will be better than a pure virgin repo with only the current snapshot, and you’ll have more confidence when you look at its history that it is all put together properly.
