Why is the performance of inference so bad?

Unity Engine
, , ,
1

I have a setup where I train 2 Agents (A and B) adversarialy. While A trains, B is in inference mode. While B trains, A is in inference mode. I train both of them for X steps and then alternate. A requests a lot more decisions than B (~500 : 1).

What I noticed is, that the performance when training A is way better than when I train B with A in inference mode.

I did some deep profiling and saw that when training A a decision request takes about 50ms. When training B, though, the decision requests from a take at least 200+ms.

Can anybody explain why the performance in inference mode is so bad?

2

Does anyone have any input on this?

3

if you use visual input is gets very slow (waiting for GPU synchronization). Also if B has much more NN then of course it will run slower . Use selfplay for the scenario you describe.

4

Thanks for the input!
I only use vector observations though. What do you mean by “NN”? In my case Agent A is my player and Agent B is an upgrade generator. A has 2 layers with 1024 hidden units and B has 2 layers with 128 hidden units.

My problem is that I have a way lower framerate when Agent A runs in inference than when it is actively learning. In my case this greatly impact the training time of B (since for that Agent A has to run in inference).

5

“My problem is that I have a way lower framerate when Agent A runs in inference than when it is actively learning” sorry idk what is the issue there.
For me Burst inference produced performance problems. “A” needs more time bc in the forward pass (multiplying out all matrixes/weights of the neurons) has more neurons so it of course needs longer.

Like I said, I would try to use Selfplay (an example is in the football players), it is specifically made to do what you want. i have never set it up so I cant help you unfortunately with details.

6

You said that A has mroe neurons so it takes longer to calculate matrices, but doesn’t it have to do this while learning too? Should the performance of A be better when running in inference vs when training?

Indeed my setup is quite similar to selfplay but selfplay only works when training a single agent. So you could let A play against itself. In my case A plays against B and vice versa so I had to implement my own version of selfplay.

7

Hi @Thorce

I believe that selfplay works even when training multiple agents. I’ve done it myself. You can do it by specifying training settings for both the agents in one training yaml file like below:

behaviors:
  agentA:
    trainer_type: ppo
    hyperparameters:
      batch_size: 2048
     ...
    network_settings:
      normalize: false
      hidden_units: 1024
      num_layers: 2
    ...
    self_play:
      save_steps: 250000
      team_change: 500000
      swap_steps: 100000
      window: 50
      play_against_latest_model_ratio: 0.5
      initial_elo: 1200.0
  agentB:
    trainer_type: ppo
    hyperparameters:
      batch_size: 2048
     ...
    network_settings:
      normalize: false
      hidden_units: 128
      num_layers: 2
    ...
    self_play:
      save_steps: 250000
      team_change: 500000
      swap_steps: 100000
      window: 50
      play_against_latest_model_ratio: 0.5
      initial_elo: 1200.0

I hope this helps.

8 
behaviors:
  Goalie:
    trainer_type: poca
    hyperparameters:
      batch_size: 2048
      buffer_size: 20480
      learning_rate: 0.0003
      beta: 0.005
      epsilon: 0.2
      lambd: 0.95
      num_epoch: 3
      learning_rate_schedule: constant
    network_settings:
      normalize: false
      hidden_units: 512
      num_layers: 2
      vis_encode_type: simple
    reward_signals:
      extrinsic:
        gamma: 0.99
        strength: 1.0
    keep_checkpoints: 5
    max_steps: 30000000
    time_horizon: 1000
    summary_freq: 10000
    self_play:
      save_steps: 50000
      team_change: 200000
      swap_steps: 1000
      window: 10
      play_against_latest_model_ratio: 0.5
      initial_elo: 1200.0
  Striker:
    trainer_type: poca
    hyperparameters:
      batch_size: 2048
      buffer_size: 20480
      learning_rate: 0.0003
      beta: 0.005
      epsilon: 0.2
      lambd: 0.95
      num_epoch: 3
      learning_rate_schedule: constant
    network_settings:
      normalize: false
      hidden_units: 512
      num_layers: 2
      vis_encode_type: simple
    reward_signals:
      extrinsic:
        gamma: 0.99
        strength: 1.0
    keep_checkpoints: 5
    max_steps: 30000000
    time_horizon: 1000
    summary_freq: 10000
    self_play:
      save_steps: 50000
      team_change: 200000
      swap_steps: 4000
      window: 10
      play_against_latest_model_ratio: 0.5
      initial_elo: 1200.0

Sorry I dont know why training is faster then inferencing for you.
@kokimitsunami
gave you a nice example how to set it up with two agents, you can also look into the yaml of the 2 strikers vs 1 goaly soccer example:

9

@kokimitsunami @GamerLordMat

Thanks for the help! Do you guys know if self play works for a non zero-sum game?

10

Maybe someone from the Unity ML-Agents team could chime in here regarding the performance problem?

11

Bump