Well, i figured out the problem
I used JsonDotNet package from the Unity Asset Store, but Remote Config wanted a different 3-d party package from git.
Turns out this Newtonsoft json from Remote Config does now support HashSet serialization. This thread solved my problem.