Hello, PlayOneShot does not use the normal AudioSource.AudioClip pipeline. If you want to use one shots, you’ll need to intercept the data in the filter chain. https://crazyminnowstudio.com/docs/salsa-lip-sync/modules/salsa/delegate-processing-custom-filter-chain/
D.
Thank you, body. It works.
I’m having a bit trouble using Salsa API, can’t figure it out from documentation
Thing is, I have my AudioSources in a Pool, all created, requested and released on demand in runtime, so there’s no single constant AudioSource in scene I could just assign to Salsa on setup phase.
So this means I’d like to leave AudioSource field as null in the component initially, and then be able to assign and re-assign (probably many times) audioSrc in runtime from code, but if I do just that, it doesn’t seem to work: lip sync animation is not played and this error keeps appearing in console:
CrazyMinnow.SALSA.QueueProcessor.Register (CrazyMinnow.SALSA.ExpressionComponent expressionComponent, System.Boolean isActivating, System.Single frac) (at <4d1a2dd8e17e48e78c476e3550c4fee4>:0)
CrazyMinnow.SALSA.Salsa.RegisterComponentOn (CrazyMinnow.SALSA.ExpressionComponent component, System.Single fracAmount) (at <4d1a2dd8e17e48e78c476e3550c4fee4>:0)
CrazyMinnow.SALSA.Salsa.TurnVisemeComponentsOn (System.Int32 triggeredIndex, System.Single fracAmount) (at <4d1a2dd8e17e48e78c476e3550c4fee4>:0)
CrazyMinnow.SALSA.Salsa.TurnVisemeOn (System.Int32 triggeredIndex, System.Single dynamicsAmount) (at <4d1a2dd8e17e48e78c476e3550c4fee4>:0)
CrazyMinnow.SALSA.Salsa.LateUpdate () (at <4d1a2dd8e17e48e78c476e3550c4fee4>:0)```
Is there a right way to do this?
Thanks!Hello, SALSA does some startup checks, so you cannot just leave the AudioSource reference as null. But you can enable the option to “wait for AudioSource”. This will bypass the checks and behind the scenes it starts a coro to monitor the reference and continue when it’s set. At this point, you can swap AudioSources at will and should not receive null-refs, AS LONG AS you leave the SALSA inspector collapsed. If it is open, it will check for the AudioSource while focused.
The null-ref your error referenced came from the QueueProcessor and wasn’t directly related to the AudioSource. However, when the AudioSource isn’t configured, it prevents the data serialization process behind the scenes and the QP doesn’t know anything about the controllers it is being asked to reference. So, putting SALSA in the “wait for AudioSource” mode allows the controller serialization to happen and all is well from that point on – as long as the Inspector is collapsed.
There are other ways to do this, like using External Analysis and letting SALSA tap your analysis as needed. Then you can control the AudioSource in your own way, but you’d need to perform and provide analysis on your own.
Hope that helps,
D
does salsa work with cc4 (character creator 4)? iclone 8? when will one clicks be available for them?
It does work with CC4. TBH, I don’t know about iClone 8. We don’t actually use/own the Reallusion products and have not had anyone ask about iClone in years.
D.
Hello,
I have a quick question on Salsa performance. I’ve been profiling my game, and I’m seeing that every few frames, Salsa’s LateUpdate is taking up a lot of time turning visemes on and off. I can’t debug the code, since it’s in the plugin, but here’s what I’m seeing in the profiler:
This LateUpdate is taking usually between 1-2 ms per character. I’m not sure if that’s just the expected overhead of Salsa, or if something unusually is happening. I’m not sure why it’s not every frame (maybe Salsa only updates visemes a few times per second?) In any case, I was curious if this was indicative of an issue, or if there was anything I could do to improve the performance.
Thanks.
Hello, this is expected behavior, SALSA polls audio analysis at intervals set by the Audio Update Delay option in the Settings tab. It looks like you have a lot of components being registered (37), so this is either multiple characters or there might be an excessive amount of components in your visemes. Each time a component is registered it cycles through all existing components in the queue to see if there are conflicts to mitigate. If you are running a single QueueProcessor for all of your characters, I would recommend against this as it creates overhead that is unnecessary. The QueueProcessor should only manage a single character since there would not be conflicts between characters. If you have 37 components per viseme, I would recommend paring that down quite a bit, less is more. Also keep in mind, registering a component compares the component to all other components currently registered for a character, including Eyes and EmoteR since there could be conflicts between those configurations. Even still your numbers seem high.
If I run a deep profile on my test environment using CC4 characters which generally have more components per viseme than most other model systems (3 - 8 per viseme), I see a 0.07ms process for registering a viseme’s components (7). If I duplicate the character with 10 instances, I see a max of 0.86ms process for registering 80 viseme components (across the 10 character configurations). These are shape and bone components in the SALSA configurations. My computer is no slouch, but it is over 4 years old (Ryzen 3900x).
Even tho your numbers seem a little high, 1-2ms is also not a serious frame chewer. 2ms is 500fps. Likely the QueueProcessor (doing all of the animation number crunching for the model) is eating a bit more. Your LateUpdate loop is sitting at <3ms. Mine is sitting at <10ms for 10 models all configured with Eyes, SALSA, and EmoteR – doing a lot of stuff – still 100fps+. I can increase to 20 CC4 models and sitting at <16ms (60fps) for the entire SALSA Suite. This sounds like a lot, but it’s 20 detailed models all independently talking, emoting, and looking at stuff – just stress testing.
If you toggle the RegisterComponentOn method, what is the breakout? It looks like you aren’t using EmoteR emphasizers which could take up more time since several emotes can be active at once (per model) depending on your configuration.
Hope this info helps!
D.
Thanks for the detailed breakdown. I’ll look at trimming down the visemes, to see how much I can get away with before it starts looking strange. To answer a couple of your questions, I’m using one QueueProcessor per character. This scene had only a single character in it. The character uses Eyes, but not EmoteR, and the total Viseme components is pretty high, as you point out:
In looking at this, I realized I just had way too many viseme configs in there that honestly didn’t need to be there. At some point, probably while I was testing out EmoteR, I found that some facial expressions didn’t look very good unless the separate Eyebrow meshes were included in the viseme config. So I made sure those were there, along with all other separate Hair meshes. But now I’m not using EmoteR, just lip sync and eye blink, so those components really weren’t doing anything.
I also realized that the model contained an LOD1 mesh with blend shapes, and those were getting picked up by the OneClick, even though that’s probably a big waste. So I got rid of those as well. I wonder if LOD awareness is something the OneClick should be made aware of? (I’m trying to decide whether there would be cases where you’d want the visemes on LOD1+ meshes. Seems just the Jaw Bone animation is “good enough” for that LOD level, at least for me.)
So now I’m down to 10 components for a the viseme that had 37 on it. Thanks.
OK, gotcha, that makes sense now. Yes CC4 can have a lot of additional meshes. As far as LOD’s, if you are configuring a single SALSA instance for all LODs, yes it will try to apply them all. It is unaware of which LOD you are on and only knows what is configured. LODs are hard to be aware of since there are so many different implementations. It might be possible to just check if a mesh is enabled (if that is the way the LOD is being switched) and only firing off registrations for enabled LOD meshes. I’ll have to look into that as an option. Let me know the performance you are seeing with the 10 vice 37. You could create a separate SALSA instance for your LODs and switch SALSA on/off with them. https://crazyminnowstudio.com/docs/salsa-lip-sync/modules/further-reading/enable-disable-modules/#enablingdisabling-salsa-suite-modules
I’ve been toying with the idea of adding a mesh inclusion array to the OneClicks that makes it more selective. It takes away the “one click” idea that way, but might be more useful considering some of these model systems.
D.
The performance was much better after trimming down from 37 components to only 10 components. Maybe down to 0.2 ms to turn visemes on and off, instead of 1-2ms each time.
As for LODs, I edited the OneClick code to look for an LOD group in the hierarchy, and to skip any renderers in LOD groups 1 or higher. But, that’s probably pretty specific to my own needs. The code’s simple, though. So at this point, everything’s working well. Thanks again for the quick and helpful response.
Hello,
I want to use Salsa in the simplest way possible in generic 2d faces, I already have the faces, I already have the different sprites mouth shapes for the phonemes, eyes open and closed. Could someone guide me on how to implement the face so that it performs the lipsync? Thank you in advance.
Also replied to your email.
We have two videos that discuss the setup and settings of SALSA.
https://www.youtube.com/watch?v=cFFD-WKraXQ
https://www.youtube.com/watch?v=7lWEWKpl0js
D.
Thanks a lot for your fast and clear answer!
I’m getting the following error when I apply my custom one click to a spawned character at runtime:
NullReferenceException: Object reference not set to an instance of an object
CrazyMinnow.SALSA.QueueProcessor.Register (CrazyMinnow.SALSA.IExpressionController controller, CrazyMinnow.SALSA.ExpressionComponent+ControlType controlType, System.Single durDelay, System.Single durOn, System.Single durHold, System.Single durOff, System.Boolean persistence, CrazyMinnow.SALSA.ExpressionComponent+ExpressionType type, CrazyMinnow.SALSA.LerpEasings+EasingType easing, CrazyMinnow.SALSA.ExpressionComponent+ExpressionHandler handler, System.Boolean animatorControlled, System.Boolean useOffset, System.Boolean isActivating, System.Single frac) (at <4d1a2dd8e17e48e78c476e3550c4fee4>:0)
CrazyMinnow.SALSA.QueueProcessor.Register (CrazyMinnow.SALSA.IExpressionController controller, CrazyMinnow.SALSA.ExpressionComponent+ControlType controlType, System.Single durOn, System.Single durHold, System.Single durOff, System.Boolean persistence, CrazyMinnow.SALSA.ExpressionComponent+ExpressionType type, CrazyMinnow.SALSA.LerpEasings+EasingType easing, CrazyMinnow.SALSA.ExpressionComponent+ExpressionHandler handler, System.Boolean animatorControlled, System.Boolean useOffset, System.Boolean isActivating, System.Single frac) (at <4d1a2dd8e17e48e78c476e3550c4fee4>:0)
CrazyMinnow.SALSA.Salsa.RegisterComponentOn (CrazyMinnow.SALSA.ExpressionComponent component, System.Single fracAmount) (at <4d1a2dd8e17e48e78c476e3550c4fee4>:0)
CrazyMinnow.SALSA.Salsa.TurnVisemeComponentsOn (System.Int32 triggeredIndex, System.Single fracAmount) (at <4d1a2dd8e17e48e78c476e3550c4fee4>:0)
CrazyMinnow.SALSA.Salsa.TurnVisemeOn (System.Int32 triggeredIndex, System.Single dynamicsAmount) (at <4d1a2dd8e17e48e78c476e3550c4fee4>:0)
CrazyMinnow.SALSA.Salsa.LateUpdate () (at <4d1a2dd8e17e48e78c476e3550c4fee4>:0)
How can I fix this?
Looks like maybe the controller data wasn’t baked. Possible you missed that step? https://crazyminnowstudio.com/docs/salsa-lip-sync/modules/further-reading/runtime-setup/#step-2-baking-controller-data
D.
I ended up writing a new script from scratch, works like a charm. Thanks for the advice.
Are there plans to have a one click for models with ARKit blend shapes or any recommended approaches? thanks
At this point, we don’t have plans to implement an ARKit OneClick – mainly because we cannot assume all models generally converted into the kit will have the same visual implementation. While this is also generally true for other model systems as well, it is particularly so with ARKit conversion. However, it is very possible if you have characters you want to implement with ARKit and your characters are standardized in naming and hierarchy, you could create your own custom OneClicks by configuring one character and using our ‘Create Custom OneClick’ add-on. You can then use the custom OneClick to apply to other models. https://crazyminnowstudio.com/docs/salsa-lip-sync/addons/custom-one-click-tool/
Hope that helps.
D.
Quick question: I’ve been noticing that occasionally, one of my characters that has an Eyes controller on it, will end up with their eyes permanently “blinked”. Their eyes stay shut, and never open again. I’ve been having a hard time diagnosing it, since it only seems to happen in a full build of my game, and it hasn’t happened in the Editor so far. I was just curious if that’s something you’ve ever seen in general?
Normally the eye blink works fine. I’ve tried a few things to resolve this, like zero-ing out all blend shapes on the model before showing them, but so far the fix has eluded me.