Cant get my ball move controller to work propely

Hi I have made a little ball controller script, but it won’t work as I wish it would.
My ball is suposed to roll when I use the directional keys, but I only want it to roll when the ball isnt rolling faster than my buttons can make it.
Can someone tell me how to solve this?

Here is my code:

using UnityEngine;
using System.Collections;

public class BallController : MonoBehaviour {

    public float ballSpeed;

    void Update()
    {
        float xSpeed = Input.GetAxis("Horizontal");
        float ySpeed = Input.GetAxis("Vertical");
               
        Rigidbody body = GetComponent<Rigidbody> ();

        if(body.angularVelocity.x >= body.angularVelocity.x * ballSpeed * Time.deltaTime)
        {
            body.AddTorque(new Vector3(xSpeed, body.angularVelocity.z, body.angularVelocity.y) * ballSpeed * Time.deltaTime);
        }

        if (body.angularVelocity.y >= body.angularVelocity.y * ballSpeed * Time.deltaTime)
        {
            body.AddTorque(new Vector3(body.angularVelocity.x, body.angularVelocity.z, ySpeed) * ballSpeed * Time.deltaTime);
        }       
    }
}

Comone this one is probably too easy for you guys but I am really stumped.
Can somebody please tell me what I am doing wrong?

I copied some of the code from a youtube video:

I changed it so it would only roll if it moves slower than my buttons can make it, but it turns out I don’t understand it good enough to make it do that.
Please can someone tell me what I need to do?