I found this:
TextMesh Pro Colors highly inaccurate - Unity Engine - Unity Discussions
Apparently it has to do with the color pickers that are all HDR, even the ones that don’t show it. I used the TMP_SDF-HDRP UNLIT Shadergraph (included in TMPro) and fed the face color directly to the output, it works there. This is not a solution (yet) though, because I might need more options than just feeding it a color in the future.
It’s still an odd behaviour, especially since UIs often follow a strict color scheme that is supposed to look the same in different situations.