3D Grid Sensor, Update for ML-Agents 2.1

Unity Engine
,
1

Hi, I’ve seen this issue come up in the forum before and also struggled with it myself - namely how to observe a variable number of objects. The grid sensor (https://github.com/Unity-Technologies/ml-agents/blob/master/com.unity.ml-agents.extensions/Runtime/Sensors/GridSensor.cs) is great for 2D, but if you need to track objects that aren’t on a plane, it’s either cameras or (lots of) raycasts.

Now, I wrote my own grid sensor a while back, because I wanted a minimal solution that does nothing more than encoding float values as visual observations. The plan was to later extend it for specific use cases. One such extension is the Spatial Grid Sensor, it detects colliders via OverlapSphere and projects their polar coordinates onto the observed grid.
I haven’t done any extensive testing yet, like comparing how well the sensor does compared to using a ton of raycasts. However, if you want to check it out, you can find the code, description and two example projects here: GitHub - mbaske/grid-sensor: Grid Sensor Components for Unity ML-Agents

4 Likes
2

Hi mbaske. This seems like a very cool extension of the GridSensor. Thanks for sharing it!

1 Like
3

I’ve updated the sensor and generalized it for 2D and 3D detection. You’ll find an in-depth description of how it works and how to use it at GitHub - mbaske/grid-sensor: Grid Sensor Components for Unity ML-Agents

I also added a comparison with the Eidos grid sensor. It’s apples and oranges, really. My sensor uses a different detection approach, which was designed for 3D and is probably a bit overkill for 2D. It’s faster though :wink:

4

My grid sensor(s) have been updated for ML-Agents 2.1 - Changes include:

  • New minimalistic GridSensorComponent and accompanying Maze Explorer example.
  • Component inspector settings are grouped into foldouts.
  • Integrated visualization and debugging options.
  • There are no longer separate 2D and 3D variants of the DetectableGameObject component.
  • Easier handling of observables, ‘Distance’ and ‘One-Hot’ are set depending on detection type.
  • Improved shape scanning algorithm and interface. Now supports concave colliders and context sensitive levels of detail.
  • The ‘Scan-At-Runtime’ option has been removed in favour of explicit method calls.
  • Frustum culling for GridSensorComponent3D with convex FOVs increases detection speed.
  • For stacked observations, the grid sensor is now being wrapped in ML-Agent’s StackingSensor.
  • Optional detectors and encoders implement the IDetector and IEncoder interfaces, rather than extending abstract classes.
  • Various classes have been renamed, e.g. the former ChannelGrid and PixelGrid are now called GridBuffer and ColorGridBuffer.
  • Code refactoring and comments.
1 Like
5

Looks very cool and well done. I’ll keep this in mind if I ever find the need

1 Like
6

Looks great - it will be next logical step for my Strandbeest training - Twitch

1 Like
7

Legend

1 Like
8

Hi I am trying to use your scripts to make a grid sensor. I have done everything as done in your DogFight scene, but I am getting the following error-
ArgumentNullException: Value cannot be null.
Parameter name: key
System.Collections.Generic.Dictionary2[TKey,TValue].FindEntry (TKey key) (at <695d1cc93cca45069c528c15c9fdd749>:0) System.Collections.Generic.Dictionary2[TKey,TValue].TryGetValue (TKey key, TValue& value) (at <695d1cc93cca45069c528c15c9fdd749>:0)
MBaske.Sensors.Grid.DetectionResult.TryGetItems (System.String tag, System.Collections.Generic.IList`1[MBaske.Sensors.Grid.DetectionResult+Item]& items) (at Assets/Scripts/Sensors/Grid/Basic/Detection/DetectionResult.cs:109)
MBaske.Sensors.Grid.GridSensorComponentBaseGO+d__22.MoveNext () (at Assets/Scripts/Sensors/Grid/GameObject/Sensor/GridSensorComponentBaseGO.cs:165)
SimpleInputControls.CollectObservations (Unity.MLAgents.Sensors.VectorSensor sensor) (at Assets/SimpleInputControls.cs:104)

Please let me know if you have any solution else let me know what other details you will require to solve this. P.S. - The error takes me here-

8654352--1164900--Screenshot (1).png
8654352--1164900--Screenshot (1).png1920×1080 250 KB

9

Hi @mohdafzal20299
I haven’t looked at this for a while… just did a clean install of Unity Editor 2021.3.151f (Windows), cloned my grid-sensor repo, re-imported the three items in /Examples/!Reimport/ folder and things look ok. I can run the example scenes without any issues.
From the error you’re getting, it looks like the ‘tag’ argument is null, causing the exception when trying to look up an associated dictionary item. Not sure where that null tag might come from though… Could you check your detectable gameobjects have valid tags and sensor settings are referencing them?