2D - Object seeping into wall

I am pretty new to Unity3D, and absolutely new to any 2D game programming in Unity. I gave it a shot by making a small platformer game in which I am a movable square. All works well, except for one thing. When I am holding down the move button and collide with a wall, the player, a square, beings to seep into the wall, and stays there until I stop moving. Both the wall and the player are using box colliders. I am not sure why my player is seeping into the wall, but I was hoping someone here might have an idea why.

This is how I move the player:

private void MovePlayer(float h) {
	transform.position += Vector3.right * h * moveSpeed * Time.deltaTime;
}

I fear my method of movement may be the issue. I tried other methods before this current method, but all of them made it so the gravity stopped working when I moved.

How can I prevent my player from seeping into the wall?
Any help is appreciated. Thanks! Here’s a screenshot:

[37370-screenshot+(24).png|37370]

apply a ridgid body and add force on keypress instead of forcing it to move?

becuase now your forcing it to move even if it is colliding.