ops.NonMaxSuppression possible issue

Unity Engine
, ,
1

Hi!

I think I’ve spot an issue with a NMS function. While thresholding by a confidence seems working properly, it looks like a IoU threshold is not taken into account. I’ve prepared a simple script to reproduce an issue:

using UnityEngine;
using Unity.Sentis;
using Unity.Sentis.Layers;

public class NMSDemo : MonoBehaviour

{
    ITensorAllocator allocator;
    Ops ops;

    void Start()
    {
        allocator = new TensorCachingAllocator();
        ops = WorkerFactory.CreateOps(BackendType.CPU, allocator);

    }

    void Update()
    {
        var x = 5f;
        var y = 5f;
        var iou_threshold = 0.6f;

        TensorFloat box_1 = new TensorFloat(new TensorShape(1, 1, 4), new float[] { x, y, x + 1.0f, y + 1.0f });
        TensorFloat box_2 = new TensorFloat(new TensorShape(1, 1, 4), new float[] { x, y, x + 1.0f, y + 1.1f });
        TensorFloat box_3 = new TensorFloat(new TensorShape(1, 1, 4), new float[] { x, y, x + 1.0f, y + 1.2f });
        TensorFloat box_4 = new TensorFloat(new TensorShape(1, 1, 4), new float[] { x, y, x + 1.0f, y + 1.5f });
        TensorFloat box_5 = new TensorFloat(new TensorShape(1, 1, 4), new float[] { x, y, x + 1.0f, y + 2.0f });

        TensorFloat[] boxes_raw = new TensorFloat[] { box_1, box_2, box_3, box_4, box_5 };

        TensorFloat boxes = ops.Concat(boxes_raw, 1) as TensorFloat;
        TensorFloat scores = new TensorFloat(new TensorShape(1, 5, 1), new float[] {0.7f, 0.6f, 0.5f, 0.5f, 0.6f});

        Debug.Log(boxes.shape);
        Debug.Log(scores.shape);

        var result = ops.NonMaxSuppression(boxes, scores, 100, iou_threshold, 0.3f, CenterPointBox.Corners);
        Debug.Log(result.shape);
        
        result.MakeReadable();
        for(var i = 0; i < result.shape[0]; i++)
        {
            Debug.Log($"{result[i, 0]}, {result[i, 1]}");
        }

        foreach(var b in boxes_raw)
        {
            b.Dispose();
        }

        boxes.Dispose();
        scores.Dispose();
        result.Dispose();

    }
}

And a equivalent python demo in notebook: colab

Despite modyfing an iou_threshold in range 0 to 1, the result tensor contains always 5 elements. The same behaviour is also with a CenterBoxPoint.Center format of boxes, and using a normalized boxes. Is it possible that I use a ops.NMS in a wrong way?

Edit: Made a mistake while copy-pasting, I’ve modified a coordinates of boxes to match a CenterPointBox.Corners sample.

2

Thanks for raising the issue.
Looks like you want Layers.CenterPointBox.Center coming from pytorch. From a glance at the implementation, iou thresholding is being used, but it could be that we’re doing something wrong. We’ll investigate a bit and get back to you asap

3

Hi,

The scores tensor is grouped by batch and class (batch, class, box). It means, that the tensor of the shape (1, 5, 1) uses a different class for each box. The boxes for different classes are not filtered by IoU. To fix your example use (1, 1, 5) instead of (1, 5, 1).

2 Likes
4

Thanks, it works.