Strange Learning Pattern Leads to Failure (NOOB)

[tomi-trescak]

Hi, I am a NOOB in RL world and I developed a simple game on which I wanted to test my skills. The game is available HERE (GitHub - tomitrescak/BaseAttackRL, Scene: BaseAttack). I have a base, which is attacked from five sides by jeeps. Your goals is to shoot all jeeps as they come:

The bullet has a “travel time” and “cadence”, so you can only shoot one bullet per second and you know if you hit something only after the bullet has reached its target. I ran the model, but results are very underwhelming. After some period of success, the soldier developed a behaviour where he shoots nothing in the corner:

Would you have time to look at that simple repo to see what am I missing?
FYI, this is my agent class:

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using Unity.MLAgents;
using Unity.MLAgents.Actuators;
using Unity.MLAgents.Sensors;

public class SoldierAgent : Agent
{
    private static Vector3 NullPosition = new Vector3(-1000, -1000, -1000);

    public JeepSpawn[] spawns;

    private Rigidbody2D rb;
    private ShootCannon cannon;
    private float angle = 0;

    [SerializeField] private float rotationSpeed = 1;


    private void Start()
    {
        this.rb = GetComponent<Rigidbody2D>();
        this.cannon = GetComponent<ShootCannon>();
    }

    public override void CollectObservations(VectorSensor sensor)
    {
        // add all spawns
        foreach (var spawn in spawns) {
            sensor.AddObservation(spawn.Jeep ? spawn.Jeep.transform.position : NullPosition);
        }
        sensor.AddObservation(transform.rotation.eulerAngles);
    }

    public override void OnActionReceived(ActionBuffers actions)
    {
        // Debug.Log(actions.ContinuousActions[0]);

        float currentAngle = -1 * actions.ContinuousActions[0] * rotationSpeed;
        angle = Mathf.Clamp(currentAngle + angle, -90, 90);
        rb.MoveRotation(angle);

        // transform.localPosition += new Vector3(moveX, 0, moveZ) * Time.deltaTime * moveSpeed;

        var shoot = actions.DiscreteActions[0];
        if (shoot == 1 && this.cannon.CanShoot)
        {
            this.cannon.Shoot();
        }
    }

    public override void WriteDiscreteActionMask(IDiscreteActionMask actionMask)
    {
        // prohibit shooting cannot during the period when it cannot be shot
        actionMask.SetActionEnabled(0, 1, this.cannon.CanShoot);
    }

    public override void Heuristic(in ActionBuffers actionsOut)
    {
        //Debug.Log("Collecting Heuristics");

        var continuoutActions = actionsOut.ContinuousActions;
        continuoutActions[0] = Input.GetAxisRaw("Horizontal");

        var discreteActions = actionsOut.DiscreteActions;
        discreteActions[0] = Input.GetKey(KeyCode.Space) ? 1 : 0;
    }


    public override void OnEpisodeBegin()
    {
        // destroy jeeps
        foreach (var b in spawns)
        {
            if (b.Jeep)
            {
                Destroy(b.Jeep);
            }
            b.enabled = false;
            b.enabled = true;
        }
        this.angle = 0;
        transform.rotation = Quaternion.identity;
    }
}

And I set rewards in the collision of Jeep with other stuff. The higher reward comes to faster moving jeeps.

using UnityEngine;

public class CarMove : MonoBehaviour
{
    [SerializeField] public float Speed = 1;
    [SerializeField] private GameObject explosion;
  
    public GameObject target;

    Transform player;
    Rigidbody2D rb;

    public JeepSpawn spawn;

    // Start is called before the first frame update
    void Start()
    {
        player = target.transform;
        rb = GetComponent<Rigidbody2D>();
        transform.up = player.position - transform.position;
        rb.velocity = transform.up * Time.deltaTime * Speed;
    }

    private void OnTriggerEnter2D(Collider2D collision)
    {

        // we shot the jeep
        if (collision.gameObject.CompareTag("Bullet"))
        {
            // Debug.Log("Bullet trigger!");

            var instance = Instantiate(explosion, transform.position, Quaternion.identity);

            spawn.Player.AddReward(this.Speed);
            spawn.enabled = true;
            spawn.Jeep = null;

            Destroy(instance, 1);
            Destroy(collision.gameObject);
            Destroy(gameObject, 0.5f);

        }
        // jeep crashed into base
        else if (collision.gameObject.CompareTag("Player"))
        {
            spawn.Player.SetReward(-100);
            spawn.Player.EndEpisode();
        }
    }
}

[JulesVerny]

Well I am pretty much a Newbie, with some very minor success with this environment.
Just some brief cursory comments, there is little to go on here:
a) Have you set up multiple environments to collect experience ? - Say 12 to 20 copies, to collect game experience across because you may need something like 5 Million Time steps across x 10 running environments ?
b) If you do have multiple environments - then all your Observational Positions will then need to be local relative, not just transform.position, since each game will be x,y,z displaced in global position. So would really need Vector3(tank.transform.position - soldier.transform.position) ** Avoid Spawning at Abosolute Positions **
c) It is not clear whether you are using Ray Cast Observations - I have found Ray Cast (limited sensor distance, so as not to interfere across running game environments) to be more effective than just delta distances, and rotations vectors.
d) It is not obvious to me how many Jeeps you spawn. But I would check progress and performance with a single Jeep, first.
e) Have you tried enabling Curiosity ? - For a very sparse reward environment, this drives the Agent to initially explore the State space, and hence perhaps easier to discover advantageous states easier. This is simply enabling an intrinsic reward and gamma, within the PPO configuration file.
f) I can see an excessive Negative reward, but the Positive Reward this.speed upon collision looks odd to me . I typically ensure that Both Positive and Negative rewards are normalised between +1.0 and -1.0. I am not sure your rewards profiles are very well normalised.

Note event when you have actually got what seems like a simple scenario operating, it can still take several hours and 10s Millions of steps for Reward profiles to actually show signs of learning and grow. RL development is a very frustrating. We only get to see the final consequences after many failures published out there.