Slowing Down When Sliding Across Walls

I have a block that has a rigidbody and a very simple script for moving around:

var speed : float = 10;

function FixedUpdate(){
    rigidbody.velocity = Vector3(Input.GetAxis("Horizontal") * speed,0,Input.GetAxis("Vertical") * speed);  
}

I also have some walls that stops the block from going off screen, but when I "Slide" across a wall, with the "wanted" velocity of the block going diagonal, the block moves much slower going across the wall than when just moving through space. This happens even when the wall has no physics material.

How do I make it so that when I slide across the wall, it moves at the same speed as when going through empty space? If there is any scripting involved in your answer, please put it in javascript.

Thanks in advance.

Problem Solved!

I was able to use Mathf.Clamp to create borders for my block, instead of actual colliders. Here is my completed script:

var speed : float = 10;
var height : float = 2;

function Update(){
    transform.position.y = height;
}

function FixedUpdate(){
    rigidbody.velocity = Vector3(Input.GetAxis("Horizontal") * speed,0,Input.GetAxis("Vertical") * speed);  
    transform.position.x = Mathf.Clamp(transform.position.x,-20,20);
    transform.position.z = Mathf.Clamp(transform.position.z,-20,20);

}

Second Solution!!!

Much better than the last, change the physics material to something frictionless. Works perfectly!