Hello,
I am trying to resume training after having exceed the max step parameter. The problem is if I increase the max-steps then the learning rate recalibrates and it messes up the whole model. Is there a simple way or a quick hack where I could have it continue beyond the max-steps while using the same learning rate decay?
I am using the linear learning rate decay.
You can use the --initialize-from feature. Rather than using --resume, specify --initialize-from= where is a previous run. It will then start a new run from timestep 0 but starting from the old model.
What will be the learning rate at timestep 0 from this new run? Will it be the lr reached at the end of the previous training run or will it start again from the value in the config file?
It will start again from the value in the config file.
What I want is for the training to resume with the learning rate from the last value