Additional observation from RaySensor

Unity Engine
, ,
1

I use RayPerceptionSensor2d. I have had some success in learning. But I have a few questions.

  1. I’m not sure, but when ray encounters a detectable object, does it go no further? Judging by Gizmos it doesn’t go any further. So I have to use 3 components of the ray sensor so that the agent sees the wall that is right behind the food?

  2. Even though I looked at https://github.com/Unity-Technologies/ml-agents/blob/08a66c4dc33340f4294643654bbe1dae74ca8ce7/com.unity.ml-agents/Runtime/Sensors/RayPerceptionSensor.cs#L164
    but didn’t really understand what kind of observations the Ray Sensor transmits to the python? I need to pass the following observations into python:

  • Distance to object.
  • The speed of the object (rigidbody2d)
  • Weight of the object (rigidbody2d)

How can I do it? I would be very grateful for code examples

2 
using UnityEngine;
using UnityEngine.Serialization;

namespace Unity.MLAgents.Sensors
{
    public class CustomRayPerceptionOutput : RayPerceptionOutput
    {
        public CustomRayPerceptionOutput()
        {
            CustomObservationSizePerRay = 3;
        }

        public override void GetCustomObservationData(RayOutput rayOutput, float[] buffer)
        {
            var rb = rayOutput.HitGameObject.GetComponent<Rigidbody>();
            if (rb != null)
            {
                buffer[0] = rb.velocity.x;
                buffer[1] = rb.velocity.z;
                buffer[2] = rb.mass;
            }
        }
    }

    /// <summary>
    /// A component for Custom 3D Ray Perception.
    /// </summary>
    [AddComponentMenu("ML Agents/Custom Ray Perception Sensor 3D", (int)MenuGroup.Sensors)]
    public class RayPerceptionSensorComponentCustom3D : RayPerceptionSensorComponent3D
    {
        public override ISensor[] CreateSensors()
        {
            var rayPerceptionInput = GetRayPerceptionInput();
            var rayPerceptionOutput = new CustomRayPerceptionOutput();

            m_RaySensor = new RayPerceptionSensor(SensorName, rayPerceptionInput, rayPerceptionOutput);

            if (ObservationStacks != 1)
            {
                var stackingSensor = new StackingSensor(m_RaySensor, ObservationStacks);
                return new ISensor[] { stackingSensor };
            }

            return new ISensor[] { m_RaySensor };
        }
    }
}

Found this on the internet, if it worked in the project it would be very cool (but it doesn’t work, no methods/fields like this). Who knows, maybe this is a new version? I don’t really want to make a custom raycast when there is already a ready-made solution

3

you should be able to get the results with
https://docs.unity3d.com/Packages/com.unity.ml-agents@2.0/api/Unity.MLAgents.Sensors.RayPerceptionOutput.html

4

And how do I do that, an example would be greatly appreciated. Now I see a solution only to make changes to the package itself, which I do not really want to do for obvious reasons

5

bump

6

you need a reference to your perception ray (can pass this like a normal reference or get it with GetComponent etc)
i.e.

[SerializeField]
RayPerceptionSensorComponent2D rayPerceptionSensor;

then in your CollectObservations function you can call something like

// Get the ray perception output from the sensor
      RayPerceptionOutput rayPerceptionOutput = rayPerceptionSensor.RaySensor.RayPerceptionOutput;
      var rayOutputs = rayPerceptionOutput.RayOutputs;

      for (int i = 0; i < rayOutputs.Length; i++)
      {
        Debug.Log($"ray {i} hit something ? {rayOutputs[i].HasHit}");
        if (rayOutputs[i].HasHit)
        {
          Debug.Log($"hit gameobject = {rayOutputs[i].HitGameObject.name}");
        }
      }

so to get the info you asked for you can do something like this

// Get the ray perception output from the sensor
      RayPerceptionOutput rayPerceptionOutput = rayPerceptionSensor.RaySensor.RayPerceptionOutput;
      RayPerceptionOutput.RayOutput[] rayOutputs = rayPerceptionOutput.RayOutputs;

      for (int i = 0; i < rayOutputs.Length; i++)
      {
        Debug.Log($"ray {i} hit something ? {rayOutputs[i].HasHit}");
        if (rayOutputs[i].HasHit)
        {
          Debug.Log($"hit gameobject = {rayOutputs[i].HitGameObject.name}");
          Debug.Log($"hit distance = {rayOutputs[i].HitFraction}");
          Rigidbody2D hitRB = rayOutputs[i].HitGameObject.GetComponentInParent<Rigidbody2D>();
          if (hitRB)
          {
            Debug.Log($"hit velo = {hitRB.velocity}");
            Debug.Log($"hit weight = {hitRB.mass}");
          }
        }
      }

(should just be a case of swapping the relevant commands to 3D if required)