I basically use a similar structure like you proposed:
I use git to version all assets, I use rsync to make a master-copy of the project available on a server - for all team members to pull copys/updates from and I maintain a master-copy with several backups on my system.
The workflow is pretty easy; we work in cycles:
cycle 1:
everybody pulls the full set of project data from the server and works with it
new assets and assets not tracked by git (e.g. graphics and models) are moved to a work-folder and later exported to a unitypackage which I import into the project, scripts, prefabs, materials, etc. are maintained with git
at the end of the cycle I update the server copy of the project and we enter cycle 2 when everybody pulls the updated project.
This works very vell so far. Sure, the manual export of asset packages are a bit of a pain, but it gives me control over the structure of the assets and ensures that no team member can mess with it.
The only real problem I have is: I maintain a perfect copy of the project on our server and after pulling from there, everyone has a exact copy on their machine… but unity needs to reimport all the assets anyway. which takes forever as the project files are way beyond 10 gb.