Let just say that my scene will have few GB of assets, not all of them will be visible at the same time.
Similar situation like in open world games.
Should I somehow manage my resources manually or unity will do all the management by itself, like removing assets that are not in use from memory.
This might be the better thing to do.
Check this out:
That aside, I’ve found Addressables a good foundation to build resource management upon - it’s the official system now if I’m not mistaken and it’s updated frequently. It has async loading and pretty much drag-and-drop grouping for any assets you’ve set an address for, to pack stuff together for download/loading and so on… switching between local vs. remote bundles is just properties. There’s a bit of a learning curve but so what, there is with everything 
…and writing the equivalent yourself would suck. Content delivery from another server works (once CORS is dealt with). Remote catalogs work (update textures or other content without needing the user to update the app etc.).
Not sure that stuff answers the question, but hope it helps.
What I want to know is - is unity going to unload some actually not used assets (textures mostly) from memory when there is not enough memory left.