You’re going to need to git rm all the files in Library and then ignore them.
You should never have been source controlling those in the first place. Be sure your source control is set up properly.
Since you have already committed large files, you probably need to just restart the repo over with the correct files.
Personally I use git (completely outside of Unity) because it is free and there are tons of tutorials out there to help you set it up as well as free places to host your repo (BitBucket, Github, Gitlab, etc.).
You can also push git repositories to other drives: thumb drives, USB drives, network drives, etc., effectively putting a complete copy of the repository there.
As far as configuring Unity to play nice with git, keep this in mind:
Here’s how I use git in one of my games, Jetpack Kurt:
Using fine-grained source control as you work to refine your engineering:
Share/Sharing source code between projects:
Setting up an appropriate .gitignore file for Unity3D:
Generally setting Unity up (includes above .gitignore concepts):
It is only simple economics that you must expend as much effort into backing it up as you feel the work is worth in the first place.
“Use source control or you will be really sad sooner or later.” - StarManta on the Unity3D forum boards