Think of your final game as a car.
A car is small. You can park it in your driveway.
Think of Unity and the Library folder especially as the car factory.
The car factory could be massive and would never fit in your driveway.
You may be able to reduce the size somewhat by eliminating unwanted packages.
Extra unwanted packages in new projects (collab, testing, rider and other junk):
About the fastest way I have found to make a project and avoid all this noise is to create the project, then as soon as you see the files appear, FORCE-STOP (hard-kill) Unity (with the Activity Manager or Task Manager), then go hand-edit the Packages/manifest.json file as outlined in the above post, then reopen Unity.
Sometimes the package system gets borked from all this unnecessary churn and requires the package cache to be cleared: