The Library folder is the partially-built project. You can’t get rid of it. BUT… you do NOT want to source control it. Its contents are merely the transient representation of your Assets, imported and ready to run at a moment’s notice.
If you are not already doing so, please consider using proper industrial-grade source control in order to guard and protect your hard-earned work.
Personally I use git (completely outside of Unity) because it is free and there are tons of tutorials out there to help you set it up as well as free places to host your repo (BitBucket, Github, Gitlab, etc.).
As far as configuring Unity to play nice with git, keep this in mind:
Here’s how I use git in one of my games, Jetpack Kurt:
Using fine-grained source control as you work to refine your engineering:
Share/Sharing source code between projects:
Setting up the right .gitignore file:
Generally setting Unity up (includes above .gitignore concepts):
It is only simple economics that you must expend as much effort into backing it up as you feel the work is worth in the first place.