Setting external rewards to multiple agents

Unity Engine
,
1

Hello there,

So far I have been using " GetComponent().SetReward(x); " to apply environment rewards to my agents, which have been controlled by PlayerAgent - I am now trying to extend the variety of agents - to, for example, a MageAgent and a WarriorAgent.

My issue now is that I would like some way to determine what kind of Agent script is attached to the current transform to apply the rewards, instead of adapting every script, such as “character stats” or “health controller” to “warrior stats”, “mage stats” etc. I believe there was a way to do this with monobehaviour scripts, but not with agents.

I stumbled upon this issue in other contexts as well, but haven’t yet found an ideal way to solve it. What would you suggest?
Thanks a lot!

2

i considered a few approaches

  • an inheritance structure (wouldnt be sure how to manage it well, though)
  • a helper object, that is static for all characters, receiving rewards → the agents then collect those rewards during their observations (which would be one step late, which might affect the learning process)
3

This is an option too I guess, a horrible one.:

if (dmger.GetComponent() != null)
{
dmger.GetComponent().SetReward(damage * rewardmodifier); //reward to attacker (dmger)
}
if (dmger.GetComponent() != null)
{
dmger.GetComponent().SetReward(damage * rewardmodifier); //reward to attacker (dmger)
}
if (dmger.GetComponent() != null)
{
dmger.GetComponent().SetReward(damage * rewardmodifier); //reward to attacker (dmger)
}

4

If it’s just a matter of scaling rewards for different agent types, then you could simply serialize the rewardmodifier field and set its values per agent instance in the inspector.

If your reward logic is more complex than that, then you can create a BasicAgent class inheriting from MLAgents.Agent and put all shared behaviour in there (Agent extends Monobehaviour btw). Agent subclasses extending BasicAgent would implement their specific reward logic. Alternatively, you could encapsulate the reward logic in a seperate object, Strategy pattern - Wikipedia

Either way, you shouldn’t call GetComponent at every agent step for performance reasons. Cache a reference to the agent during initilization and use that instead.

5

Sorry, the rewards were a bit misleading, thanks a bunch though, I believe your answer put me on the right track.

It was all about the issue of getting access to the agent methods regardless of varying scripts.

Currently trying to implement this n launch a training.
I’ll have to refactor afterwards, but to keep it as simple as possible i just made one class with BasicAgent : Agent and let the other (3 so far) inherit from that as suggested - now simply calling .GetComponent.AddReward()

I had earlier tried to call GetComponent to achieve this, which wouldn’t work - I’m not exactly sure why, though.

→ I guess the way i have it implemented now assumes that there is only one agent class attached to the transform?
I am wondering this because i once had in mind to implement a set of abilities with inheritance, so that, for example, abilities could be assigned to a character or agent, which then would call “UseAbility” or so on the parent class and trigger whatever ability is assigned to the transform - what would happen if there were multiple abilities assigned to the transform, though?

Having a look at the caching, i feel like I’m using GetComponent<> way too often in my project - I didn’t quite figure this one out yet :stuck_out_tongue:

Thanks again!

6

That sounds like you would want to use composition rather than inheritance. Like defining an IAbility interface and then assign concrete Ability instances to your agents. Either through code during runtime, or perhaps with scriptable objects, something like this https://dev.to/eriksk/implementing-the-strategy-design-pattern-using-scriptable-objects-in-unity-292i

7

Thanks, I have stumbled upon the scriptable objects once, but believe i got stuck at some point and decided to revert to independent classes, which worked for that purpose - yet, it is certainly something i want to redo in the next project :slight_smile:
Thanks as well for the link :slight_smile:

Live long and prosper!
Greetings,
Sephraih