Rigidbody and player passing through it, only if there is multiple rigidbodies

I have a wall, that usually can’t be passed through, until I start adding other walls to the scene. The closer to the other objects, the more the player goes into the wall (see below)

I am using this code to move my player (in a fixedUpdate() function)

moveDirection = new Vector3(speed * Input.GetAxis("Horizontal"),0, speed *     Input.GetAxis("Vertical"));
moveDirection = transform.TransformDirection(moveDirection);
this.transform.rigidbody.velocity = moveDirection;

My player and wall both are rigidbodies with collision detection at continuous. Eventually my player just passes right through the wall. Any help? I have tried to look, but none of the topics were exactly like mine

One solution that will most likely fix the problem is decreasing the FixedTimeStep (Edit->ProjectSettings->Time). Try 0.01 for instance. It’s hard to say precisely what the problem is from just the code and small pics you provided.