Saved model

Why did it save the model before finishing the training? Is this normal?

2020-06-24 20:03:42 INFO [stats.py:100] Pruebath128v2_CubeLearning: Step: 2700000. Time Elapsed: 5059.919 s Mean Reward: -8.231. Std of Reward: 12.789. Training.
2020-06-24 20:12:52 INFO [stats.py:100] Pruebath128v2_CubeLearning: Step: 3000000. Time Elapsed: 5609.333 s Mean Reward: -7.589. Std of Reward: 12.291. Training.
2020-06-24 20:22:13 INFO [stats.py:100] Pruebath128v2_CubeLearning: Step: 3300000. Time Elapsed: 6170.680 s Mean Reward: -5.681. Std of Reward: 8.657. Training.
2020-06-24 20:31:01 INFO [trainer_controller.py:112] Saved Model
2020-06-24 20:31:41 INFO [stats.py:100] Pruebath128v2_CubeLearning: Step: 3600000. Time Elapsed: 6738.388 s Mean Reward: -62.983. Std of Reward: 89.105. Training.
2020-06-24 20:41:08 INFO [stats.py:100] Pruebath128v2_CubeLearning: Step: 3900000. Time Elapsed: 7305.699 s Mean Reward: -41.377. Std of Reward: 40.700. Training.
2020-06-24 20:51:15 INFO [stats.py:100] Pruebath128v2_CubeLearning: Step: 4200000. Time Elapsed: 7912.293 s Mean Reward: -10.417. Std of Reward: 12.720. Training.

mlagents-learn periodically checkpoints the model, so if the program crashes or the process gets otherwise interrupted, you can use --resume to resume training.

1 Like

Thx!