By default, there’s no x for the default button class list. If you click “Add Style Class to List”, it will enable you to remove the styles. However, the class will automatically be re-added arbitrarily.
Can you clarify what version of Unity you’re using and if you’re using the UI Builder from package or not?
What you’re describing is sounding confusing to me, as default classes should not be able to be removed in the first place.