My high school used my class and the one after it as a testing bed for a new math program. After the class after me graduated they dropped the course…
So I’m pretty horrible at math. I mean, I use Khan Academy to brush up my skills when the wife is away and my head is too foggy to code. I still struggle on some of the early high school and middle school math. haha.
I think the MOST important thing is knowing of things. You don’t need to have stuff memorized or be great at it. Just know what vectors are, the basics of geometry (basics…) and trigonometry. When I was programming games in PyGame with Python I had to know a lot of math. The cool spiral effect for my missile had to be done entirely with math. However with Unity you can use simple vector addition with a start and end location, and animate the spiral. 
There are times when I facepalm because something complicated I did had a much simpler mathematic solution, but really it hasn’t happened that often to me. So the ratio of time spent in school (I have 0 college btw) vs the ratio of time spent fixing a mistake I made because I didn’t go to school would be incredibly high. So, my advice is actually to just save your money and learn on your own. Set a learning schedule for yourself where you set aside study time on relevant subjects, and watch some open education lectures, of which there are plenty.
Now, if you really are going to school and aren’t to be persuaded otherwise, I would be decent with math. Not that you’ll need it all for game dev, but the exercises you’ll get will likely require it.
One last thing to point out. There’s a huge different between designing your systems (Like writing all your own shaders, voxel games, procedural designs, new game engines, etc) then you’re going to need a crap ton of math skills, ya. You’ll meet a lot of people though that will tell you to not do those things though. I’ve been programming for about 4 years, doing game development for about 2 of those, and my goal is to do some of those things eventually. However, in the mean time I’m creating fun, challenging games that are rewarding that I can play right now, and learning the needed skills for my long term goals as I go. With this method I haven’t had the huge stress of school, job, wife, projects and deciding whether or not I want to sleep that day or not.