Adapt the reward function during the training, best way to do it ?

AI & Machine Learning ML Agents
1

Hi,

I would like to adapt the reward function during the training. What is the best way to do that ?

My function is a sum of different weighted terms: by changing the weights I would like to focus slowly to hardest exercices. It is a kind of curriculum learning but how can I get the level of the curriculum in the code ?

Thanks,

2

You could define curriculum environment parameters and hook your reward logic to their values.

3

Curriculum training works if the custom “parameters” is a simplistic model. For example in the wall jump example adjusting the height of the wall as the agent improves.

However if you are using a complex simulation with a complex shaped reward and in my case over 40 parameters then instead of curriculum training what you could do is handle these parameters in your own code and adjust them based on the performance of the agent.
Take a look at Agent.GetCumulativeReward() at the end of each episode to monitor the performance of the agent. If you have multiple agent instances (as you should) then calculate the moving average over a suitable time window.
Agent.StepCount
will give you the number of steps the agent has executed within each episode. Resets to zero at the end of each episode. So capture this at the end of each episode and keep a running total.
Note: you cannot use Academy.Instance.StepCount for the same thing because by default (Unless you have modified it) the academy steps on every fixed timestep.

However: If you are doing this you have to be careful not to adjust the parameters too often. Take a look at the Buffer size in the training config yaml file. If you are tweaking the parameters more frequently than at least 10 or so buffer sizes of steps it will not turn out well.

In my case if I have a buffer size of 25000 steps. I allow at least 500K steps before allowing parameters to be changed. Otherwise the model will be trying to hit a moving target that is moving way faster than it can possibly learn.

Also take a look at Academy.Instance.StatsRecorder
For recording your own metrics.
Example for Averaged Scalars:
Academy.Instance.StatsRecorder.Add(@“ZNPC/Social”, Stats_Score_Socialize, StatAggregationMethod.Average);
Academy.Instance.StatsRecorder.Add(@“ZNPC/SomethingElse”, Stats_Score_SomethingElse, StatAggregationMethod.Average);

Or for Histograms:
Academy.Instance.StatsRecorder.Add(@“ZAgent/PartialSolve”, Stats_RewardSolvedPartial, StatAggregationMethod.Histogram)

Beware. The agent neural networks are very sensitive. So the smallest inconsistency or bug in custom code will wreck havoc on training results.

1 Like