Anyone successfully used ml-agents visual observations with first-person gameplay?

Unity Engine
, ,
1

Hi all,

I was wondering if anyone could share some success stories, or not-a-success-stories, using visual observations in a first person agent perspective with ml-agents? e.g., a first person ml-agents shooter?
I’m mostly seeing ray-casting-sensor solutions. Currently doing research before undertaking this in ml-agents.

Any specific challenges, interesting solutions, or other things worth sharing?

There’s a couple of papers out there on the topic, but so far I haven’t seen many ml-agents projects on this yet.

2

Have you been looking into the Obstacle Tower Challenge? This was a while back, but if I remember correctly, people were successfully using ML-Agents with visual observations to solve it.
Announcing the Obstacle Tower Challenge winners and open source release | Unity Blog
GitHub - Unity-Technologies/obstacle-tower-env: Obstacle Tower Environment

1 Like
3

Obstacle Tower in its vanilla state is a 3rd person task. It can be easily configured to run in first person view.

I’d suggest to look into literature that utilizes FPS environments, like Doom or this one. These sources could provide you with ideas on how long the training takes and what hyperparameters might be suitable. This knowledge should get you started on training an FPS agent using ml-agents.

1 Like