As a general rule of thumb, I just pay attention to art assets/materials/etc., source code, prefabs, and scenes, and let Unity do whatever it wants with the rest.
But if you really want to go around deleting files in a project you care about, make sure you are using a VCS. If you have External Version Control enabled, and are using something like Git or SVN, everything in the Library folder (includes both cache and previews) can be completely ignored. As described here External Version Control Documentation. Ignoring Temp is always a save bet too, Unity should dump this every time it closes.
If you are using Git, you can add Library/* and Temp/* to your .gitignore file. You want to keep the .meta files versioned. They keep versioning clean and allow for cross platform development.
Once you have everything important committed (and everything else ignored), if you are really concerned about automatically generated files taking up space, you can clone your project and let Unity rebuild what it needs. Know though, that deleting auto generated file can result in having to reimport some or all of your assets, which can take time.