Hello all! I’m very excited to announce NatML, the machine learning runtime for Unity Engine. NatML is a high performance, cross-platform machine learning runtime for Unity. With NatML, you can run your ONNX machine learning models across iOS, Android, macOS, and Windows. With NatML, you are guaranteed to have bare metal performance, as NatML takes advantage of hardware accelerators like the Apple Neural Engine.
NatML is very (very) easy to use. It has the best runtime performance for machine learning, often multiple times faster than Unity’s Barracuda. It is has full support for ONNX, and doesn’t require writing custom code to convert models for compatibility (much unlike Barracuda). And it has tight integration with other NatSuite Framework API’s like NatDevice and NatCorder, giving you the ability to create highly interactive ML-powered apps in very few lines of code.
NatML Features:
High Performance Machine Learning. NatML takes advantage of the Apple Neural Engine on iOS and macOS (CoreML); the Neural Processing Unit on Android (NNAPI); and the GPU on Windows (DirectML). See the NatML benchmark against Unity Barracuda.
Cross Platform Machine Learning. You can run your ML models on device, in the Editor, and anywhere in between because NatML supports iOS, Android, macOS, and Windows.
Realtime Computer Vision. Using NatML with NatDevice, you can create apps that do realtime object classification (see the classifier example using MobileNet); realtime object detection (see the detection example using YOLO); and much more. Face detection and pose detection examples coming soon!
Full Layer Coverage. NatML supports the full ONNX specification, which means you can use ONNX models from any ML framework (PyTorch, Tensorflow, others) or from anywhere. No need to write conversion code.
Community Tools. We are developing an open-source extension library which will provide out-of-the-box support for making easy predictions with more models, all from our developer community.
We also partner with studios to quickly and affordably create custom models that can then be used with NatML. For this and any other questions:
Iam using this asset to develop an educational sound based game , the sample scene has one dummy audio clip to test which return its equivalent generated text however when I try to record my own audio clip through microphone using Unity’s Api and send that clip for prediction , it returns an empty string every time. Please help me how can I fix it.
Hey there, I presume you are using the Silero STT predictor package? If so, then the model requires a specific audio format for the audio data. As a result, I recommend using NatDevice to stream the microphone with the required format. See the example code that uses NatDevice.
Hello there. I am trying to use the MobileNet v2. But i am having some issues. After having imported the MobileNet v2 unity package and the NatML - Machine Learning Runtime unity package, I am met by a wall of errors, all variants of:
Assets\ML\mobilenet-v2\Runtime\MobileNetv2Predictor.cs(10,23): error CS0234: The type or namespace name 'Extensions' does not exist in the namespace 'NatSuite.ML' (are you missing an assembly reference?)
Can you tell me what I am doing wrong?
Have a great one.
I saw an example where the ML model is pulled from the NatML hub in code. If we purchase a model such as MoveNet from the hub can the model get embedded in Unity application vs downloading from the hub at run-time.
NatML Hub is designed to only deliver models to users at runtime. There are two reasons for this:
NatML doesn’t actually require running a specific ML model format (CoreML vs. TFLite vs. ONNX) and NatML Hub takes advantage of this to provide an ML model optimized specifically for the device that requests the model. You can read more here.
In addition to the model graph, Hub also delivers supplementary data for the model including classification labels (for classification and detection models), normalization coefficients, and more.
Assets\ML\mobilenet-v2\Runtime\MobileNetv2Predictor.cs(10,23): error CS0234: The type or namespace name ‘Extensions’ does not exist in the namespace ‘NatSuite.ML’ (are you missing an assembly reference?)
Assets\ML\mobilenet-v2\Runtime\MobileNetv2Predictor.cs(11,23): error CS0234: The type or namespace name ‘Features’ does not exist in the namespace ‘NatSuite.ML’ (are you missing an assembly reference?)
Assets\ML\mobilenet-v2\Runtime\MobileNetv2Predictor.cs(12,23): error CS0234: The type or namespace name ‘Internal’ does not exist in the namespace ‘NatSuite.ML’ (are you missing an assembly reference?)
Assets\ML\mobilenet-v2\Runtime\MobileNetv2Predictor.cs(18,48): error CS0246: The type or namespace name ‘IMLPredictor<>’ could not be found (are you missing a using directive or an assembly reference?)
…ext
I just installed it with the latest version and the following problems appeared. I installed it on a project I previously installed opencv .
Can NatML and Barracuda exist in the same project? After importing NatML, the NNModels that I had were no longer recognized as NNModels by Unity. Once I deleted NatML, they worked again.
Hi! I am using unity 2020.3.24 and when using the sample tiny-yolov3 sample it crashes when playing in editor when it gets to this line:
IntPtr[ ] IMLModel.Predict (params IntPtr[ ] inputs) => (this as MLEdgeModel).Predict(inputs); in the MLModel.cs script.
Any idea what could it be?
Its just the dog image of sample project.
Hey there, thanks for bringing this to our attention. I recommend using other object detection models, like yolox, ssd-lite, or nanodet. We’ve archived our current tiny-yolo-v3 model, and will be replacing it with yolo-v5.