I can answer your question precisely. I freakin’ HATE fiddling and clicking with FBX export, etc.
Keeping files ONLY as Blender files is awesome, one of the most endearing simplifying things about Unity with Blender.
But as you intuit above, there are potential hidden costs. Here are the two major costs.
Cost #1: every place you wish to build this will REQUIRE Blender be installed. Your computer? Fine! Install it! Your friend’s computer? Eh, maybe you can entice him into installing it. Now… Unity’s cloud build? Not so much. In fact, you will never see Blender installed by anybody offering cloud build / SaaS type services for the simple reason that it falls outside of Blender’s licensing.
Cost #2: you become beholden to changes in: Blender export behaviour, or Unity export script. The former is relatively stable, but does change from time to time. The latter can also change, such as around about 2018 or so Unity changed their python export script to suddenly include Camera and Light models by default. This broke all previous imported models, which ignored Camera / Light. If you failed to remove the Light/Camera from your scene, now your hierarchy moves around, you have an extra Camera and Light, and if you made prefabs from the mesh, those are broken now.
I have elected to keep using Blender, but I have introduced a wart into my development process: every time I install a new version of Unity3D I go into the Python export script that it installs and I hack it to NOT import Camera/Light, since there will NEVER be a time that I want a Blender Camera in my scene.
That change Unity made in the import script to make it start importing Camera and Light was really in my opinion completely useless to 99.9% of simple model-and-import workflows. The only possible place for such a feature might be for importing a huge complete scene with lights and cameras, and even that is useless since everything will need to be tweaked in Unity anyway!
Here’s some more of my mad high scribblings about this:
Unity imports Blender3D objects as FBX via a little Python script:
The Python script that Unity uses (substitute your Unity version number or search) to import: