[Released] Awesome ML Kit — Optimized real-time ML Models centered around computer vision

[leavittx]

Awesome ML Kit gives the Unity developers easy way to employ optimized real-time ML models centered around computer vision.

The current set of models for the Unity Barracuda inference library includes

1. High-quality background removal/background matting based on RobustVideoMatting (optimized for Unity)
2. Depth estimation by single image
   And more is planned!

Features

1. Sample scene for each model
2. Specially made visualizations with different post effects
3. Select solid color / image / video as background replacement
4. Different model versions for Quality / Speed tradeoff
5. WebCamTexture support
6. Unity VideoPlayer integration
7. AVPro Live Camera integration (recommended)

Ideal for

• Media installations
• Apps with video streaming (e.g. can be used with Agora.io)
• Kiosks

Performance
NVidia RTX3070

• Background removal: 300 FPS
• Depth estimation: 400 FPS

NVidia GTX1050Ti (mobile)

• Background removal: 50 FPS
• Depth estimation: 80 FPS

Awesome ML Kit 2. Background removal and webcam virtual background in Unity

To show what’s possible to do with such asset, we’ve created a remix of the Disclosure - Hourglass ft. LION BABE music video where all the male dance masking is done using this asset (we just discard the color information from the webcam)

Awesome ML Kit X Disclosure - Hourglass ft. LION BABE

Also Demolition Media Hap was used in the creation process of this experience (to sync with original video)

Note: only desktop platforms currently supported (tested on Windows DX11 and Linux Vulkan)
Unity versions: 2021/2022 and above

Get it on the Asset Store: Awesome ML Kit | AI-ML Integration | Unity Asset Store

[leavittx]

A small update for all existing and new users: in most cases the Gamma color space should be selected in the project settings in order for the colors to be correct on the video. You can do that with Edit->Project Settings->Player->Other Settings->Color Space set to Gamma

[leavittx]

We got a big update recently which is covered in this video:

Awesome ML Kit 2. Background removal and webcam virtual background in Unity

[DeJMan]

1. The link to the documentation is a google doc which is editable by all users.
2. The link to AVPROLive integration in section 2.3 is an older version of the same file which is linked at the start. I dont know if this is intended.
3. For AVPROLive, the only documentation to set it up is your link to the example files… which don’t work and show missing scripts. Edit: Fixed by deleting the AVPROLiveUI gameobject.
4. After a build, the output shows up as pink. No errors in the logs. Edit: Fixed pink by deleting the universal render pipeline dependency which your asset needs for some reason.
   

[leavittx]

@DeJMan

1. The link to the documentation is a google doc which is editable by all users.
2. The link to AVPROLive integration in section 2.3 is an older version of the same file which is linked at the start. I dont know if this is intended.
3. For AVPROLive, the only documentation to set it up is your link to the example files… which don’t work and show missing scripts. Edit: Fixed by deleting the AVPROLiveUI gameobject.
4. After a build, the output shows up as pink. No errors in the logs. Edit: Fixed pink by deleting the universal render pipeline dependency which your asset needs for some reason.

Hi. Thanks for the info.

1. Fixed
2. Fixed
3. Tested in an empty project (Unity 2022, latest AVPro Live Camera) - got no missing scripts. That’s how it should look like in the first example scene:
   
   The missing script may be manually re-added as shown above, if the UI is needed. Sometimes Unity script references are quite fragile when distributed in my experience.
4. Docs section 1.2 says that URP is required for the example scenes (however 1st and 2nd scenes might work without it, but that’s not recommended). The asset itself doesn’t require URP and the model inference can be done in the built-in or in any other scriptable RP.