Questions regarding YOLOV8n on HuggingFace and Functional methods

Unity Engine
, , , ,
1

Hi there,

Inspired from the YOLOv8n example provided on HuggingFace, I have the following code for a task I am trying to achieve:

// Get the indices
var scores = Functional.ReduceMax(allScores, 0) - 0.7f;
var classIDs = Functional.ArgMax(allScores, 0);
var indices = Functional.NMS(boxCoords, scores, 0.3f);

// Get the mask
var selectProtos = Functional.Select(Protos, 1, indices); // indices -> no. of dets after NMS, protos -> (32,8400)

How do we handle the case when the indices variable has 0 detections? This would result in the shape [0] and when used in the Functional.Select() function, throws the following error for real-time inference:

IndexOutOfRangeException: Index was outside the bounds of the array.
Unity.Sentis.Layers.Select.Execute (Unity.Sentis.ExecutionContext ctx) (at ./Library/PackageCache/com.unity.sentis@1.5.0-pre.3/Runtime/Core/Layers/Layer.Transformation.cs:977)
Unity.Sentis.GenericWorker.Execute () (at ./Library/PackageCache/com.unity.sentis@1.5.0-pre.3/Runtime/Core/Backends/GenericWorker.cs:133)
Unity.Sentis.GenericWorker.Execute (Unity.Sentis.Tensor input) (at ./Library/PackageCache/com.unity.sentis@1.5.0-pre.3/Runtime/Core/Backends/GenericWorker.cs:105)

It is possible that the YOLOv8n does not detect anything in the scene, and since we can’t use an If operator to check the shape of the FunctionalTensor within Functional implementations, how should one go about fixing this?
Also please recommend what is the best way to approach such problems, where you need to select the best detection index based on some operations within the method, for example max bounding box height?

Cheers

2

Bump!

Also had a follow-up question regarding the steps prior to NMS:

The outputs from the network result in 8400 detections. Instead of using all the 8400 detections for NMS, I would like to filter out the detections first based on a threshold. So initially, I would want to discard all the detections having a score less than 0.5, allowing me to increase the efficiency of NMS. However, I am yet to find a method within Functional class that can help me do so, I have looked at TopK, but didn’t find that useful. Would be great to have a method that could result in a FunctionalTensor with the indices above 0.5 score. ArgMax results in the best index, but it would be better to choose say for example the top K indices.

Please let me know if there is another way to achieve this via the Functional class.

Thanks in advance.

3 
var indicesPassing = NonZero(scores > scoreThreshold);
var boxesPassing = IndexSelect(boxes, -2, indicesPassing);
var scoresPassing = IndexSelect(scores, -1, indicesPassing);
var nmsIndicesPassing = NMS(boxesPassing, scoresPassing.Reshape(new[] { 1, 1, -1 }), iouThreshold).Select(1, 2);
var nmsIndices = Gather(indicesPassing, 1, nmsIndicesPassing.Unsqueeze(0));
return Squeeze(nmsIndices, new[] { 0 });

this works, try it out. We’ll make it more elegant in a future release

1 Like
4

Just tried it out! Works great! However, I switched to an end2end ONNX model and stitched the NMS part at the end of the model to transform it into a pipeline that includes post-processing as well. Really excited for future releases!