Using Ray Perception Sensor Component 3D, how would I get the following rays set up: ForwardLeft, Forward, ForwardRight + ForwardDown, Down + Back? As in this image:
And also, just to confirm, I don’t need to add the ray count to the Behavior Parameters → Vector Observation → Stack Size anymore, but it’s now somehow managed automatically in the background, right?
And a side question, how could I have the debug gizmo ray visualization show outside of training, while still in non-play mode in editor? I can’t see any lines when setting up this component, making it hard to understand what’s happening.
Hi,
You should be able to do this setup if you’re using ML-Agents 0.13.0 or higher. In that version, we added a “Use Child Sensors” option to the Agent, so it will look for SensorComponents (e.g. Ray Perception sensors) on GameObjects nested below the Agent in the hierarchy.
So you should need:
a RayPerceptionSensorComponent3D attached to the Agent’s GameObject with RaysPerDirection=1 and MaxRayDegrees=45 for the forward, forward-left, and forward right rays
a GameObject attached to the Agent’s GameObject with a transform to point it backwards, and a RayPerceptionSensorComponent3D attached to this with RaysPerDirection=0 (will still shoot 1 ray) for the backwards ray
for the forward-down ray, either do similar to #2, or attach to the Agent’s GameObject with a negative End Vertical Offset
Correct.
Good question, I don’t think we can currently do this, but I think there is some options for gizmos to render even outside of play mode. I’ll look into this for a future version.
Implementing this now! I wonder if for learning and usability sake, it would make sense to rename RaysPerDirection to say RaysPerSideDirection or something. Considering that forward also is a direction, and right now, RaysPerDirection=1 gives me 3 rays in your example, and RaysPerDirection=0 gives me 1 ray. (Maybe there’s also something that could be done to have it easier understandable that RayPerceptionComponent is basically forward-facing by default, it wasn’t immediately clear to me. In Unity code, such would usually be very clear via e.g. “transform.forward * foobar”.)
Either way, loving ML-Agents, and I don’t have a deep understanding of the framework so please ignore this if it doesn’t make sense.
@celion_unity
Hello. It’s a different issue from the one above, but I have a question, so I’m writing a comment. I’m currently working on a 2D project, and when I used ray permception2D in that project, it’s hard to detect properly because I keep hitting my own collider. Can you tell me what I should do when I don’t want to detect a collider that I don’t want?
For your information, I added the following inspector value.
Sensor Name->RayPerceptionSensor
Detectable Tags->player(that’s not my agent’s tag name now, but I want to use this tag for my agent)
Ray Per Direction → 5
Max Ray Degrees → 70
Sphere Cast Radius → 0.1
Ray Layer Mask → EveryThing
Observation Stacks → 1
Use World Position → true
Hi @jinhueck ,
I think the 2D Physics raycasts behave a bit differently that the 3D ones by default. The simplest fix should be to go to Project Settings > Physics2D and uncheck “Queries Start In Colliders”, but beware that this will affect all queries, not just the ML-Agents ones.
If that’s too big of a change, I think you can either
Use the transformed GameObject approach described above, so that the rays for the SensorComponent start outside of the Agent’s collider
Add a specific layer to the Agent’s collider, and use the Ray Layer Mask to ignore that layer
Quick question regarding the Ray Perception Sensor Component. I get the setup of the ray perception sensor component in the editor. Now how do I “React” to the raycast hit infos that arise from those rays hitting things. What is the Event name? OnCollisionEnter ? OnTriggerEnter? OnRayCastHit ? Nowhere can I find this vital bit of information.Please put me out of my misery
A way to access the collision info associated with raycasts is not directly accessible through our API. Typically, we let the agent learn how to ‘react’ to the data captured by the raycasts. Can you describe what you’re trying to do and maybe we can offer an alternative?
If you have game logic that depends on getting this info, you can try something similar to how we implement the laser mechanic in our FoodCollector agent.
So, I understand that child ray perception sensors are “automatic” in the sense that no code is required for agents to use them and learn/act accordingly. You don’t add them as observations, and you don’t account for them in the vector observation space size, correct?
Yes, this is correct. I tried to make this point in the section in the documentation about sensors, but if you have suggestions on how to make it clearer, I can try to add them.
I see “RayPerceptionSensorComponent - Allows information from set of ray-casts to be used as observation.” in that documentation.Is it possible to turn RaycastHit into an observation without using Ray Perception Sensor?
I made a custom “eye” for my agent with lots of raycasts, but I can’t get it to work with ML agents. Another idea I have is to connect a Vector3(a hit point) and an int(use a number to define a type of obejct, walls can be 0 and buttons is 1) so the agent can learn to avoid walls and find buttons. Is this possible?
