Why do my Physics2D Settings Reset on Test

Whenever I test my scene, my physics2D settings are ignored and are replaced with something else. Specifically my layer collisions matrix. This only happens in-game. In the editor, all my settings are perfectly fine. If I want to test something, I have to make change to my matrix by hand every time I want to test.

Forgive me if I’m missing something here, but what is causing this? How do I fix it?

Thanks.

After changing your Physics2D settings, save your project.(menu: File → Save Project)