Where to find info/examples on adding additional data (categories?) to a classification model?

You will have to forgive me if I use the wrong terminology, or misunderstand some processes/workflows here, as a lot of this is new to me.

I grabbed the handwriting recognition demo and wanted to use that as a basis to test out other things, so I grabbed a model named ‘inception’, exposed a Texture2D and added some images and wanted to see if I could make it work, which I was able, for the most part.

I am not exactly sure that what I am wanting to do is as easy as I think it is, but I believe it is. It is my understanding that I can take a model, a bunch of images of ‘something’ along with a label of what that ‘something’ is and provide it to the model for additional training so that it is then able to detect said thing?

I have a Google Coral, an Intel NCS2, and a Jetson Nano, as well as a proxmox Win10 VM with a RTX2070 (I mention these in case having one of them is necessary for what I am trying to accomplish. I needed the Nano to run DeepStack, but don’t know if they are needed for actual training, or just the actual runtime detection stuff, and my PC is linux with an AMD card, so no Cuda).

I mostly have those devices for my home automation system and doing object detection via my camera system. This is what lead me to believe that I should be able to do what I am wanting to do, as my setup for my camera system lets me upload photos of things/people, and their name, and then the camera system would be able to recognize that thing or person specifically. I am just not sure how it does that under the hood.

Unfortunately, due to my lack of overall knowledge in this field, I am not quite sure what to search for in order to try and find code examples of this process.

So I suppose, what it comes down to is, does anyone know where I might find a simple and straight-to-the-point example of how one might add additional detections to a classification/recognition model? The language used in the example is not as much of a concern as having a full working example. If I have that, I will be good to go, I just am not sure what I should be looking for.

If anyone is able to point me in the right direction, I would be most appreciative.
Thanks,
-MH

I suppose I will try and list my findings and results here, so if there are any other ground-zero beginners with this kind of stuff, they may find the information useful.

I was able to find a dotnet example that actually works on linux. (a few didn’t due to having UI that was using Windows APIs)

Now, it works, I can see the results when I run it, but it seems to include what looks like a Protobuf binary named inception, which I am assuming is the model, just not in the onnx format that Unity is using or that is additional information used along with the inception model? If so, I am not sure where it’s getting the model currently, as it was not specifically included, unless it was downloaded via Nuget in one of the ML packages and I didn’t realize that.

So I suppose the next step is figuring out how the results I am seeing in this project can translate into a context more akin to Unitys, using an onnx model. I suppose it is worth a check to see if some of these API’s used in this project are available within Sentis, and can be done as-is in Unity, without having to do it externally first. That is probably what I will check now.

-MH

Would this help?
https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html

1 Like

From your post my understanding that what you want to do is “transfer learn” a pretrained model.
As you want to make new classes classifiable the last layer of the model must be replaced.

This link may be useful Google Colab

You don’t have to do that in C#. It’s probably easier to do that in Python as more material is available on deep learning. You just have to convert the model to ONNX after completing the model.

3 Likes

Appreciated. I was able to get it figured out for the most part last night. At least, I was able to train an inceptionv3 model on a few categories, export it to onnx, and use it with Sentis. I am still not 100% there yet, as it only exported/converted with the 3 categories I was doing my testing with. None of the original classifications (I think 1000 or so were originally there). At least it’s a solid start, I probably just missed something in the set/settings.

I will be sure to take a look at what you provided, thanks. :+1:

1 Like

My pleasure :slight_smile:
If you want to keep the existing classes, you need to include those classes in the training data too. And the last layer would have an output size of old classes + new classes → 1003 classes. If you replaced the last layer with a new one, that layer will be trained from scratch which is why you need that original data.

So you could take a subset of the data the original model was trained on and include your custom data in that.

1 Like