I have the same thing happening to my VRChat avatar’s materials when i try to change the material at slot #4.
I don’t know how to fix it, as it’s most likely an internal Unity bug, that is present in both the version that VRC uses (2018.4.20f1), and the one that ChillourVR uses (2019.4.13f1), and even the newest 2021.1.0f1 version.
edit: I’ve reported this bug yesterday, today i got a response that they know about it and are tracking it here: Unity Issue Tracker - [LowHangingFruit] Material is applied to two slots when applying Material to a single slot while recording Animation
(The issue describes something slightly different than i’ve reported, but it seems to stem from the same bug)
So if you want to incentivize them to fix it faster: vote on it 
However, working around it is quite straightforward:
Stop changing the material at slot #4.
Open your model in Blender/other modelling software, and on the list of materials move one that you don’t need to change to slot #4 (in Blender it seems they’re not numbered, remember it’s the 5th one). Then export it back to FBX/whatever (preferably to what it was originally, i suppose), adjust your material-changing animations in Unity, and voila.
If this will be your first time exporting this model to FBX/whatever, try to make sure you get all units and transforms the same as the author of the original model (the skinned mesh (‘body’) and the armature (skeleton) can be rotated or have a scale of 100 instead of 1 for example) so you won’t have any issues later on.
Or skip that part, and just fix it later, once any issues emerge.
Now, if you want to change all materials at some point, then instead of just moving them around, it seems that you need to add a new material (that won’t be actually visible anywhere in VRC) and move that one to slot #4. Of course that will use up a slot that VRC counts and in some cases may make your avatar move to a worse performance rank, but it’s not that big of a deal IMO.
Hope this helps.