My character jumps occationally when against a wall?

My FPS character controller seems to jump up when being pushed against other colliders, I have checked and it is not the step offset or the slope limit. It does not happen every time, but every now and then when walking along the wall. What else could be causing this problem? The walls with the colliders are straight and not at an angle, is it just a problem with the character controller itself?

Well, I know I had asked this a while ago, but I did find the answer to it eventually, I just never thought to come back and post about it, but I might as well post the answer to anyone else who might need help.

I had accidentally put multiple colliders on a plane created in Unity 4, now the planes created in unity are not just one poly, they are 10x10, so like _MGB_ was asking, if I was just using a mesh collider on a plane created in Unity it would have caused that error.

However, this was not the case, instead I had placed both a box collider, and somehow a cloth collider onto it, now this was before Unity 5 of course and there were still 2 different types of cloth colliders, I was using a static one. In any case, it didn’t help and in that I found my issue.

Maybe the terrain has bumps?