Hello! I’m trying to create a game where the agent act as the enemy and choose the attack based on their attack effectiveness. So, here’s my code:
using System;
using System.Collections;
using System.Collections.Generic;
using Unity.MLAgents;
using Unity.MLAgents.Actuators;
using Unity.MLAgents.Sensors;
using Unity.MLAgents.Sensors.Reflection;
using UnityEngine;
public class EnemyAIAgent : Agent
{
public float attackPower = 10f;
public float weaknessMultiplier = 2f;
public float resistanceMultiplier = 0.5f;
public float blockMultiplier = 0.2f;
private int NUM_ELEMENT;
private float reward = 0f;
public AttackScript attackScript;
void Start()
{
attackScript = GetComponent<AttackScript>();
}
public override void OnEpisodeBegin()
{
RequestDecision();
// RequestAction();
}
public override void CollectObservations(VectorSensor sensor)
{
// Observe the attack element
NUM_ELEMENT = (int)AttackScript.magicElement.LastElement;
sensor.AddObservation(NUM_ELEMENT);
}
public void AgentAttack(ActionSegment<int> act)
{
FighterStats fighterStats = GetComponent<FighterStats>();
FighterAction fighterAction = GetComponent<FighterAction>();
var dirToGo = Vector3.zero;
var rotateDir = Vector3.zero;
var physicalAttack = act[0];
var fireAttack = act[1];
if(physicalAttack == 1)
{
attackScript = GameObject.Find("EMeleePrefab").GetComponent<AttackScript>();
fighterAction.SelectAttack("melee");
Debug.Log("Melee attack");
} else if (fireAttack == 1)
{
attackScript = GameObject.Find("EFireMagicPrefab").GetComponent<AttackScript>();
fighterAction.SelectAttack("Fire");
Debug.Log("Fire attack");
}
if (physicalAttack == 1 || fireAttack == 1)
{
if (attackScript.IsBlockingAttack)
{
// reward = -attackPower * blockMultiplier;
reward = -1f;
} else if (attackScript.IsResistingAttack)
{
// reward = -attackPower * resistanceMultiplier;
reward = -0.5f;
} else if (attackScript.IsWeakToAttack)
{
// reward = attackPower * weaknessMultiplier;
reward = 1f;
} else
{
// reward = attackPower;
reward = 0.5f;
}
}
}
public override void OnActionReceived(ActionBuffers actions)
{
AgentAttack(actions.DiscreteActions);
// Apply the reward
AddReward(reward);
Debug.Log("Reward: " + reward);
// NextTurn();
EndEpisode();
}
}
So, I notice my CollectObservation method seem not work because when I call RequestDecision on my GameController, the agent does nothing. Any suggestion about what I should do?
so what makes you say collectobservation isn’t doing anything? is it being executed?
do you have 2 discrete branches of size 2?
1 Like
Well CollectObservation method is executed after I try, i try debug my NUM_ELEMENT. I have 2 discrete branches of size 1. Gosh, it really worked out when the size is 2, but I have two follow up question:
- Why should the size be 2 not 1? Because my if method just use 1 instead of 2 (e.g if physicalAtttack == 1, fireAttack == 1)
- My mana is being quickly depleted to 0. Is this a bug or because the ML Agents still run too quickly? Even I use time-scale = 1 when i run ML Agents. This is also my curiosity because when I use Debug.Log at NUM_ELEMENT, the log came up endlessly.
arrays are 0 indexed so branch of 1 is going to always output 0, so if your ‘if’ statements are checking for 1 they’ll never be true until you make them branch size of 2 (now the outputs are 0 or 1)
at timescale 1 ML will run in fixed update so 50 frames in 1second, you are constantly ending and starting the episode in each step so it will run the same logic 50 times in 1 second, if you want the agent to run less you need to call requestdecision less often
1 Like
Thank you very much for your explanation, I want to ask one last question (hopefully so), I tried to use Academy.Instance.AutomaticSteppingEnabled = false;, to limit the step, but i think it still the problem because I called the EnvironmentStep at FixedUpdate, can I reduce the 50 frame or somehting to limit it so it called once?
Here is my full code:
using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using UnityEngine.UI;
using System.Transactions;
using UnityEngine.SocialPlatforms;
using Unity.MLAgents;
using Unity.MLAgents.Actuators;
using Unity.MLAgents.Sensors;
public class GameController : MonoBehaviour
{
private List<FighterStats> fighterStats;
private GameObject battleMenu;
public Text battleText;
private void Awake()
{
battleMenu = GameObject.Find("ActionMenu");
}
void Start()
{
Academy.Instance.AutomaticSteppingEnabled = false;
fighterStats = new List<FighterStats>();
GameObject hero = GameObject.FindGameObjectWithTag("Hero");
FighterStats currentFighterStats = hero.GetComponent<FighterStats>();
currentFighterStats.CalculateNextTurn(0);
fighterStats.Add(currentFighterStats);
GameObject enemy = GameObject.FindGameObjectWithTag("Enemy");
FighterStats currentEnemyStats = enemy.GetComponent<FighterStats>();
currentEnemyStats.CalculateNextTurn(0);
fighterStats.Add(currentEnemyStats);
fighterStats.Sort();
NextTurn();
}
private void FixedUpdate()
{
Academy.Instance.AutomaticSteppingEnabled = false;
Academy.Instance.EnvironmentStep();
}
public void NextTurn()
{
battleText.gameObject.SetActive(false);
FighterStats currentFighterStats = fighterStats[0];
fighterStats.Remove(currentFighterStats);
if (!currentFighterStats.GetDead())
{
GameObject currentUnit = currentFighterStats.gameObject;
currentFighterStats.CalculateNextTurn(currentFighterStats.nextActTurn);
Debug.Log("NextActTurn: " + currentFighterStats.nextActTurn);
fighterStats.Add(currentFighterStats);
fighterStats.Sort();
if (currentUnit.CompareTag("Hero"))
{
battleMenu.SetActive(true);
Debug.Log("Hero's turn");
}
else
{
battleMenu.SetActive(false);
Debug.Log("Enemy's turn");
// Scripted AI
// string attackType = Random.Range(0, 2) == 1 ? "melee" : "range";
// currentUnit.GetComponent<FighterAction>().SelectAttack(attackType);
// ML-Agents AI
// float[] actions = new float[2];
// float[] actions = new float[] { 0, 1, 0, 0, 0, 0, 0 };
// ActionBuffers actionBuffers = ActionBuffers.FromDiscreteActions(actions);
// currentUnit.GetComponent<EnemyAIAgent>().OnActionReceived(actionBuffers);
currentUnit.GetComponent<EnemyAIAgent>().RequestDecision();
}
}
else
{
NextTurn();
}
}
}
simplest method is perhaps removing the manual calling of request decision / environmentstep and using the decision requester component as that lets you add a custom delay (if you need more control then you can code your own)
p.s. you have an infinite loop of calling NextTurn when the agent is dead
1 Like
Thank you very much for your answer, I’m experimenting between using RequestDecision method or DecisionRequester component.
And I appreaciate your notice about my infinite loop, I’m currently fixing it.