Just happened to have seen this feature mentioned in the 'Here’s what’s new in Unity 2021.2’ video on Unity’s youtube channel and by chance I’m working on project that could benefit from such a feature so figured I’d give it a spin.
Firstly I’m really pleased to see Unity committing resources to these sort of endeavours, especially the virtual camera and face tracking. Whilst not every developer might use them I think in time they will become an essential tool for developing 3D software as they open up so many possibilities. I’m also impressed with what I’ve seen and experienced so far with the virtual camera and its app - all very professional. The ‘Getting Started guide’ was very useful and I was able to get up and running within minutes.
Most of what I’ll discuss here is in direct relation to my current project, so I figured I’d provide a brief outline first. I’d been tasked with providing extensive amount of video of dancers recorded using MS Kinect and rendered using Unity VFX. The finally video will be projection mapped onto a building behind a live band playing a 1 hour set of music to which these virtual dancers are dancing too. I figure having a virtual camera might simplify the process of animating the camera in the scene.
Edit:
Slight problem, it seems installing the Live Capture package in 2021.2.8f1 causes an instant hard crash when trying to load RenderDoc. Confirmed this happens several times, even after deleting the Library and eventually creating a totally new project no additional code just default core HDRP 3D packages. Once I removed the package from my project I was able to load RenderDoc again.
Uploading a bug report: Case 1402654
Question - When will this be available for Builds?
Will the virtual camera and/or face tracking be supported in builds? Ultimately I feel this will be essential and the product is only half complete without. While currently I could use the virtual camera to pre-record my camera animation, it would open up a whole new avenue to be able to deliver a build of the project to my client and let them direct the camera.
Question - Why are there so many camera type components?
So my only real confusion in Unity comes from the many different components all sharing various camera settings that are similar or the same, yet apparently are not shared.
I’m also confused as to why when I created a Virtual Camera Actor it didn’t automatically clone my main camera settings or give me the option to do so.
It seems odd to me that the VirtualCameraDevice has its own camera settings for lens, camera body and isn’t using the Virtual Camera Actor camera settings. More so that the VirtualCameraActor has its own lens, intrinsics, camera body and then that gameObject also a has a Unity Camera with its own Lens, camera body, yet none of these settings are synced or shared.
I guess there must be a good reason for this, but in my tests I simply set everything to be the same and that worked well. Maybe there could be a feature to clone the settings between all these camera types?
There are also a large number of ‘professional’ camera settings here that i’m just not used to and don’t really want or use. For simpler projects it would be nice to have a simple component that automatically deals with these.
Question - Playback hooks
For my use case I’m going to be recording the camera motion around animating objects ( vfx point cloud from Kinect ). Assuming I can find some way to trigger the kinect playback, does the Take Recorder have any hooks or events that I can use to detect when it starts playing a take?
Comments - The App
Reset Pose Button
Generally this looks and works great. My only real gripe is that the Reset Pose button needs to be much more prominent and have a better icon. It took me way to long to discover it and its pretty much essential in order to start viewing the scene in the right place. I’m not sure I like it being combined with the reset lens option either.
Had I used the help system I might have found it sooner, but I’d also rather an auto popup help system that went through all the main controls instead of you having to click on them ( though that’s great for later exploration). Ultimately I feel such a button should have been discoverable through use of the app instead of it being tucked away to the side.
Tracking Loss
The other issue I ran into which stumped me for a while was losing tracking, upon which I seem to be randomly re-positioned in the scene. I eventually worked out that having cleared out my workspace for recording with the Kinect i’d removed any trackable features from most of the walls. So when I was focusing on a virtual object in the scene, but got too close to a wall it lost tracking.
I’d like to see a pop-up in the app for situations where this happens as I can see it being very confusing for some users and it was quite jarring.
Comments - Unity
Streaming - Not working
It appears if you don’t have the game view active you will not get an active stream on the device, or maybe there is but its just black. It took me a while to discover why the app didn’t appear to be streaming when I had the scene view active and game view was not. Its a bit weird though as I thought I started in game view and streaming wasn’t working, so switched to scene view to check the virtual camera motion, so maybe something else went wrong.
Streaming - Performance
Performance wasn’t great, but that’s likely a combination of elements, such as old hardware. I was also testing in edit mode as the ‘getting started pdf’ seemed to suggest, but i’m sure I read elsewhere that in edit mode you might get more hitches.
I tried changing the video settings and reduced my game view to 1280x720, but it still hitched.
I tried using NVENC H.264 but it just gave an ‘InvalidOperationException: Encoder failed during initialization.’ error. Having a 3060TI i’m pretty sure this is supported on my hardware and I’m sure I’m using that setting in other software ( not running at the same time ) without an issue. The error log didn’t provide any other useful details so not sure what I can do to debug this.
Server Problem.
Generally I find as long as I don’t ‘stop’ the server in Unity the app connects fine. However as soon as I stop the server and then restart it, it doesn’t matter what I do on the PC or on the ipad I cannot reconnect and the app can’t even discover the server! The only way to get things working again is to remove the server and create a new one.
Well thats it for my first foray into the virtual camera. Like I said it seems very impressive and generally well designed, though a little confusing for the lay person. I look forward to play around with it more whenI have time, and seeing if I can actually use it for my current project.