Tensorboard metrics data missing

Unity Engine
,
1

Hello!

For my thesis, I want to evaluate the training metrics myself. For this I use the events.out.tfevents. … . … . … .0 files. I have encountered the problem that some data points are missing in some metrics.
I want get an average for cumulative reward, episode length, and Elo for two training runs. I have loaded the files and got the metric keys. As you can see in the screenshot, the list mean_reward has two tuples of size 6000, but mean_lenght and mean_elo have two tuples with different tuple sizes. Because of this, I can’t create a ndarray to calculate the functions I want.
I don’t know what can cause the problem and what would be the best solution.

I really appreciate any help.

7388546--901928--tuple_sizes.PNG

2

It is possible that if the summary frequency is too high, there are summaries that do not see an episode ending at all. Increasing the time between summaries (in the config.yaml) should help. You need to set it to a value such that at least one group episode ends for each summary.
Alternatively, the data points in tensorboard should indicate number of steps. You could drop the data for which not all metrics are available for a particular step.