Help with FPS agents.

Unity Engine
,
1

Hi,

I’m trying to train some AI in a fps game using multi-agents. My level is a small square area.

I am struggling to get them trained as they either just spin around(Mostly this one), get stuck on walls or just walk around aimlessly. The AI are rewarded for hitting an enemy and also for killing an enemy. Teams are also rewarded for winning. The AI at the moment are not punished for dying so there is no reason for them to stop and hide behind objects. The mean rewards seem quite low, even when ive changed things to improve rewards and encourage more hitting, it has improved but the AI dont act as expected. I’ve tried both only rewarding for a win + increasing reward with kills, and also doing this plus punishing for a lose. Both seem to not work. The rewards just seem very low and not improving, or if it does, it drops straight back down.

This training environment is 5 maps being played at once. I was wondering am I not rewarding the agents enough? Have I just left it for not long enough? Is their potential that the map is too complicated for the AI, but the AI do manage to shoot the enemy sometimes and they do manage to sometimes stand by each other. Or is there something I’ve missed.

behaviors:

behaviors:
  Shooter:
    trainer_type: poca
    hyperparameters:
      batch_size: 2048
      buffer_size: 20480
      learning_rate: 0.0003
      beta: 0.005
      epsilon: 0.2
      lambd: 0.95
      num_epoch: 3
      learning_rate_schedule: constant
    network_settings:
      normalize: false
      hidden_units: 512
      num_layers: 3
      vis_encode_type: simple
      goal_conditioning_type: none
    reward_signals:
      extrinsic:
        gamma: 0.999
        strength: 1.0
    keep_checkpoints: 40
    checkpoint_interval: 2000000
    max_steps: 500000000
    time_horizon: 1000
    summary_freq: 50000
    threaded: false
    self_play:
      save_steps: 500000
      team_change: 1000000
      swap_steps: 200000
      window: 100
      play_against_latest_model_ratio: 0.5
      initial_elo: 1200.0

8392419--1107606--upload_2022-8-26_13-14-16.png
8392419--1107606--upload_2022-8-26_13-14-16.png1189×645 54.1 KB

8392419--1107609--upload_2022-8-26_13-14-46.png
8392419--1107609--upload_2022-8-26_13-14-46.png1259×849 103 KB

8392419--1107612--upload_2022-8-26_13-15-10.png

Thank you very much!

2

8392668--1107630--upload_2022-8-26_14-34-17.png
8392668--1107630--upload_2022-8-26_14-34-17.png920×645 88.5 KB

Forgot to add to the main thread.
Here is a list of observations that I have

3

Just wanted to see if there any more information i could provide to get help to get my AI working as intended :slight_smile:

4

How long have you run the training set for? I’m building a model that is uses a quasi-realistic physics model to steer an aircraft around a giant cube and it spends the first 5-6 million steps just getting the ‘lay of the land’ (i.e., not faceplanting immediately due to gravity).

If you find your agent experiencing mode collapse (e.g. in my case, just flying around in circles), that might be an indicator that your model isn’t complex enough to capture the behaviour you want to replicate (which means either more units or more layers).

You might consider starting off with getting the model working on simpler tasks like just getting agents to walk to a random point in the map with a small penalty each step to get it to hurry up. (And a hard reset if it gets stuck or tries looping). Then you could teach it to use cover by adding static turrets through the map and then eventually add agent vs agent combat.

5

Thank you this has helped, i also realised in comparison to a project like the dodgeball one I have been only training it for 1/6 of theirs so i am going to also let it continue to train much longer.