About Sentis Beta

Welcome

Welcome to the Sentis beta group! Here you can find all information related to the Sentis beta program, share feedback, ask for help, and have discussions about Sentis. If you are brand-new to Sentis as the neural engine in Unity, this quick (2m) video will help get you up to speed on what it does:

If you are looking for a deeper explanation and some rich examples of how Sentis can be used, you can watch our longer (45m) Unite talk:

About

Sentis is a neural inference engine for AI models that leverages the compute on end-user devices, instead of a cloud server. It can run anywhere the Unity runtime can be deployed. It replaces Barracuda and addresses much of the feedback we received while Barracuda was in an early prototype phase. Most importantly, Sentis can run many neural networks performantly at game speed. Sentis is currently in an open beta phase (“pre-release” package) with more improvements and iterations coming. We plan to launch Sentis to production (“release” package) with Unity 6 (AKA: Unity 2023 LTS) around October 2024. Sentis is free for all Unity users.

Sentis allows you to import AI/ML neural network models (from sources like TensorFlow, PyTorch, and other AI hubs) in the ONNX format into the Unity Editor, then optimize and run inferences in real-time on user devices across all Unity supported platforms.

You can do this in the Unity Runtime (for gameplay) or in the Unity Editor (for game creation) instead of requiring a cloud server with an expensive GPU. Sentis is fast, avoids cloud costs, and allows for private neural network inferences. Ultimately, Sentis empowers Unity developers to take advantage of the incredible innovation happening in AI that was not previously possible. There are many possible use cases and we are excited to see what you create during the beta. Here are a few examples we have seen to get you brainstorming:

1. Image Upscaling: Upscale a low-resolution image, model, texture, etc.
2. Style Transfer: Translate the look or image effect of a scene to a new style
3. Non-Playable Characters (NPCs): Automate the dialogue of characters
4. Speech Recognition (NLP): Interpret live speech
5. Body/Object Detection: Detect objects using the camera
6. Depth estimation: Detect depth using the camera
7. Image, Video, and 3D Model Classification: Detect on-screen objects
8. Handwriting Classification: Detect handwritten letters, symbols, or numbers
9. Creating one-of-a-kind player experiences: Generate infinitely unique gameplay scenarios
10. Animation: Apply automated pose rig to a rigid body
11. Simulation helper: Approximate a complicated function with a neural network to drive gameplay logic
12. Timeseries: Anomaly detection, forecasting/prediction

Getting Started

Install the Unity Sentis Package

  1. Download Unity 6 (2023.x) from the Hub. Sentis 1.3.0 works with 2023.2+, while prior versions of Sentis work with Unity 2021.3+.
  2. Create a new Unity project or open an existing one.
  3. Click Sentis Beta Pre-Release package, or go to Window > Package Manager, then click the + icon, select “Add package by name…” and type “com.unity.sentis”
    sentis-3
  4. Press the Add button to install the package.

    Note: For versions 1.0.0 and 1.1.0, Sentis was in a closed beta and required an authorization key. For version 1.2.0 and beyond, Sentis will not require a key.

Resources
General product info
Documentation
Project samples
Hugging Face models
Roadmap
Blog posts

Reporting bugs
Please report bugs via the Unity Bug Reporter in the Unity Editor, and provide as much context as possible so we can rapidly triage the issue. When submitting a bug, please:

  • Link to a hosted image or video that clearly shows or explains the issue, including expected vs. actual results.
  • Attach a (stripped) project so we can easily recreate your issue.
  • Attach Profiler .data and Profile Analyzer .pdata files where possible.
  • After filing your bug, open a Discussions topic if you believe the community would benefit. In the topic name, it helps to use your bug name and issue # to make it easier to identify
Posting Topics

Please keep your new topics very specific and try to ensure you are not duplicating an existing topic. Here are some topic themes to guide you:

  1. Features and workflows: Are there any workflows that are unclear or missing? Are there any features that you expected to see, but didn’t? Can you share a clever workaround for an issue that you think may help others? Feel free to also schedule a live meeting with us if you would like to discuss your idea in detail.
  2. Ease of use: Is it easy for you to use Sentis even if you may not be an AI/ML expert?
  3. Performance: Does it perform well with a wide variety of neural network model types? Are there any performance issues on platform/OS combinations?
  4. Use cases: What are your use cases and what neural network models are you using? Can you share a video demo of what you’re working on? We understand models, games, and other assets may be proprietary, so please do not share any sensitive intellectual property (IP) publicly.
  5. Bugs: Is there a bug that you reported through the bug reporter that’s worth discussing with the community? (See above for specifics about submitting bugs to Unity directly.)
  6. Documentation: Is any of the documentation unclear? Would you like more documentation for specific items?
Frequently Asked Questions

Where can I find neural network models to run in Sentis?
Here is a list of some of the most popular sources for neural networks. Hugging Face, Akira, Caffe Model Zoo, Facebook, Google, Keras, ModelPlace io, Model Zoo / MXNet, ONNX Zoo, OpenAI / Microsoft, PyTorch Hub, TensorFlow Hub.

When will Sentis be promoted to a “release” package (aka production, aka general availability)?
Sentis is currently in beta as a “pre-release” package. We plan to promote it to “release” status later in 2024 around October, when Unity 6 launches (aka Unity 2023 LTS).

How do I get specific or private help that I don’t want to post about on the public forums?
Reach out here here to collaborate or get your game/app featured.

Policies

Please comply with the basic policies we have set up to ensure a smooth beta program for you and for Unity.

  1. Prototyping & private distribution only: The beta versions of this package are for prototyping and private distribution only, and your access is subject to our Terms of Service (TOS). This helps protect your work from breaking changes that can occur during the beta cycle. In particular, please review section “13. Evaluation Versions” in our TOS and ensure you understand and can comply with these terms before continuing in the program. If you would like to ship a commercial game/app with Sentis before it becomes a “release” package, then you can reach out to us to get a one-page addendum to our TOS if your legal team requires it.

  2. Code of Conduct: The general Code of Conduct applies to Sentis on Unity Discussions. Please familiarize yourself with it to ensure productive and inclusive conversations.

MuseCube_small Thank you for your excitement and interest in AI @ Unity!

77 Likes

@Sentis-closed-beta-users we are excited to announce a package update for Sentis v1.1.0 that is now live in the package manager. The update is mostly focused on improved performance for importing and running AI models, and the improvements are 100% based on user feedback from this beta program. Thank you all very kindly for providing very specific feedback based on your use cases and models. You can find what’s new here and the quick upgrade guide here.

For users who have already downloaded the Sentis package, your update to v1.1.0 should happen automatically. For users who have not installed the Sentis package yet, v1.1.0 will be the first version that is installed by default.

:pray: Please continue all the great feedback!

:heart_eyes: Thanks,
The Sentis Team

15 Likes

Hi all, we are happy to release an update to Sentis today, version 1.3.0-pre.3. It is a minor version release to address some user-reported issues and bugs. Below are the release notes and here’s a link to the latest docs. Also, a reminder that we have a nice growing repo of project samples here that can help you get started quickly with real-world use cases.

Added

  • Improved documentation clarity in some cases
  • LoadModelDesc method in ModelLoader now public
  • TensorInt data type for Clip
  • Docs page for exporting models to ONNX format
  • Model sources in documentation

Changed

  • clearOnInit default to false in tensor data pinning
  • NonZero asserts on command buffer as unsupported
  • Improved inspector for custom layers and allow horizontal scrolling
  • Rewrote docs and sample code for model execution in parts

Fixed

  • Thread group count for Split
  • Importer no longer fails when the inspector is out of focus
  • Docstrings for accuracy
  • Broken links in documentation to ONNX model zoo
  • Dense import step with transpose
  • Warnings for async methods in tensor data classes

Removed

  • Broken Dense to ScaleBias optimization
3 Likes

:fire: We’re excited to announce that we recently released Sentis 1.4. It’s a big update that includes huge API improvements and performance boosts. Below is a summary, and the docs contain the detailed changes and upgrade guide.

Important Note: Some of these are breaking changes and that means that you must reexport previously serialized .sentis files and encrypted serialized models using Sentis 1.4.

1.4 summary:

  • Functional API: We have introduced a new functional API for building, editing, and chaining models that is intuitive, stable, and optimized for inference. The functional API supports PyTorch-style tensor methods, operators, sliced indexing, and type promotion while remaining 100% compatible with your existing ONNX models.
  • Backend API: For applications that require lower-level, lowest-overhead manipulations of tensors, you can use the Backend API instead of the Functional API. This is meant for experienced users who want to fully control memory management and scheduling. However, it does not come with usability or safety checks.
  • Operator support: We now support even more ONNX operators.
  • Quantization: We now support model weight quantization (FP16 or UINT8) in the Unity Editor if you want to reduce your model size by up to 75%.
  • Scheduling time: We reworked model scheduling so it’s 2x faster, fixed the memory leaks, and made sure our Garbage Collection is as close to 0 as possible.
  • Smart serialization: Our new FlatBuffer-based serialization format improves the security, speed, and portability of serialized Sentis models. A model that has been serialized in Sentis 1.4 and later will no longer be tied to a specific Sentis version; it can be imported in future package versions.

We’d love your input on what we should build next, so please vote on our public roadmap. Or, reach out directly to collaborate, get help, or get your game/app featured!

Cheers,
The Sentis Team

5 Likes

Hi all and @Sentis-closed-beta-users, I’m excited to tell you about our v1.6.0 release that we launched yesterday. This release is focused on adding new ops and improved inference performance, based on your feedback and awesome projects. We hope you find it useful!

Added

  • Support for GridSample ONNX operator along with functional API method.
  • Functional API method for numpy-style RandomChoice.
  • BitonicSort backend method on GPUCompute for a fast GPU sort.
  • Improved error handling when loading unsupported legacy .sentis models.

Updated

  • Fully rewritten NonMaxSuppression operator, with fast inference on CPU and GPUCompute backends.
  • Reduced CPU allocation in some operators, resulting in less garbage collection.
  • Fixed inference errors in for some operators such as Slice and Multinomial.
  • Optimized inference for some Gather operations with a single index.
  • Optimized loading of compute functions for better performance on GPU.
  • Unified how tensors of zero length are handled reducing potential errors.
  • New methods for downloading and cloning tensors to the CPU.

We’re getting close to the production release of Sentis coming with Unity 6 in mid-October, and we are excited to start seeing more projects being shipped publicly with Sentis all the time. As always, you can check out our high-level roadmap here. If you have a new big idea, we’d love it if you submitted it through that portal.

Cheers!

2 Likes