Time Scale and physics problem

My game contains photomode, where you can stop time. The problem is after I stop the time physics act incorrectly. I mean when some object where going down after bringing time to normal they suprisely are going up or other stupid direction. Why unity acts like that? Have you any idea how can I fix the problem?

Here is simple photo Mode controller:

void Update()
{

    if (Input.GetKeyDown(KeyCode.RightArrow))
    {
        timeFactor += 0.05f;

        Time.timeScale = Mathf.Clamp(timeFactor, 0f, 1f);
        Time.fixedDeltaTime = Time.timeScale * .02f;

        timeTxt.text = Time.timeScale + "x";
    }

    if (Input.GetKeyDown(KeyCode.LeftArrow))
    {
        timeFactor -= 0.05f;

        Time.timeScale = Mathf.Clamp(timeFactor, 0f, 1f);
        Time.fixedDeltaTime = Time.timeScale * .02f;

        timeTxt.text = Time.timeScale + "x";
    }

    if (Input.GetKeyDown(KeyCode.Space))
    {
        timeFactor = 0;

        Time.timeScale = timeFactor;         
        Time.fixedDeltaTime = Time.timeScale * .02f;

        timeTxt.text = Time.timeScale + "x";
    }
}

I don’t think you should set “Time.fixedDeltaTime” all the time. It controls the rate of FixedUpdate() and you don’t want to change that, leave it at 0.2 or set it to 0.1 but you are not supposed to change it every frame that you are doing now…

I copy-pasted this into a scene with a bunch of falling cubes and couldn’t replicate the issue - stopping time made the cubes stop falling, and resuming time made them continue falling at the expected rate.

Can you provide a short video clip or something showing the incorrect behavior? And by any chance is there anything else in your scene that affects Time.fixedDeltaTime?