IEnumerator cuts time in half everytime it is called

Hello, I am trying to get a Image.fillAmount to set from 1 to 0 with a IEnumerator. It works the first time it is called but when I call it a second time, the time at which the Image.fillAmount is set is cut in half.

The first time it is called it takes 1 second to finish,
The second time its called it takes 0.5 second to finish
The third time is even quicker.

IEnumerator SwordSwingCooldownTimer()
    {
        float rate = Time.time * swordSwingCooldownTimer;
        float progress = 0.0f;

        while (progress < 1f)
        {
            swordSwingCooldownImage.fillAmount -= rate;
            progress += rate * Time.deltaTime;
            yield return null;
        }
    }

Possibly I am doing something wrong with the float rate, I’ve tried Time.deltaTime and Time.time.

float progress = 0.0f;

         while (progress < swordSwingCooldownTimer)
         {
             progress += Time.deltaTime;
             swordSwingCooldownImage.fillAmount = progress / swordSwingCooldownTimer;
             yield return null;
         }

Something like this maybe?