max_step with multiple agents

Unity Engine
, ,
1

I remember reading something a while ago, either on this forum or in the documentation, that you need to times the max step of each agent by the number of agents in the scene.

Is this correct? and why?

2

Hi @StewedHarry ,
In what context to do you want to do this? I’m not sure what you’re trying to accomplish. Could you elaborate?

3

I’m trying to train an agent with other agents in the scene running on inference. I remember reading somewhere that either the max step or step count need to be adjusted to account for multiple agents in the scene. However, I can’t find this in the documentation.

4

So there is a confusing variety of similar things ml-agents calls ‘steps’. (I’m still figuring out all the relationships too so please correct me if any of these are wrong)(as of release_6)

  • Physics step - Called one time on each agent per fixed timestep using FixedUpdate()
  • Agent step - The same as a physics step, although potentially different because it can be incremented independently. Corresponds to an atomic change of the engine that happens between Agent decisions. Accessible using agentObject.StepCount (will get only that agent’s current step count within the current episode)
  • Agent Max Step - The most steps an agent can possibly take before being automatically reset and starting a new episode, agent steps in an episode may be less than this if the agent is reset earlier via code (for example falling of the platform). Accessible using agentObject.MaxStep
  • Academy (Environment) steps - Total steps by all agents within unity are accessible using Academy.Instance.TotalStepCount or for all steps only in the current episode Academy.Instance.StepCount

With that understanding I don’t see a reason you would need to multiply max step by the number of agents unless you were trying to figure out how often a policy update occurs or something.

5

Max Step on the Agent will inform the agent of how many steps it can take in an episode. The summary values output by the python process are the addition of all of the agent steps in the current environments. So when you see a summary at 2000 steps, for example, that 2000 steps was calculated by adding all of the steps of each agent in the active environments together.

In short, you should not have to do any complex math on the agent max step based on the number of agents.

6

Ok great, I think I understand. Thanks for all the info.

I wish I could remember where I came across the multiplication of step by the number of
agents - it was one of the developers working on the ML-Agents.