How do I make the player's mouth move whenever they speak so other players can see it?

Unity Engine
, ,
1

I have already created the functionality for detecting when the player speaks and move the mouth with it. But how do I sync it over the network with netcode? I tried ServerRpc/ClientRpc and this which was the closest to working but it only works on Host because ConnectedClients can be only used on Server.

private void UpdateMouth()
    {
        foreach (var networkClient in NetworkManager.Singleton.ConnectedClients)
        {

            if (networkClient.Key == 0)
            {
                if (MicLoudness*100 <= 1f && endTalking)
                {
           
                    //FixMouthServerRpc();
                    return;
                }

                endTalking = false;
                mouth1.Value = networkClient.Value.PlayerObject.GetComponentInChildren<SkinnedMeshRenderer>()
                    .GetBlendShapeWeight(1);
               
                networkClient.Value.PlayerObject.GetComponentInChildren<SkinnedMeshRenderer>().SetBlendShapeWeight(1,
                    _skinnedMeshRenderer.GetBlendShapeWeight(1) + Time.deltaTime * reverseMouth);
                if (networkClient.Value.PlayerObject.GetComponentInChildren<SkinnedMeshRenderer>().GetBlendShapeWeight(1) <= 0f)
                {
                    reverseMouth = -1f * reverseMouth;
                    networkClient.Value.PlayerObject.GetComponentInChildren<SkinnedMeshRenderer>().SetBlendShapeWeight(1, 0.01f);
                }
       
                if (networkClient.Value.PlayerObject.GetComponentInChildren<SkinnedMeshRenderer>().GetBlendShapeWeight(1) >= 100f)
                {
                    reverseMouth = -1f * reverseMouth;
                    networkClient.Value.PlayerObject.GetComponentInChildren<SkinnedMeshRenderer>().SetBlendShapeWeight(1, 99f);
                }
                StartCoroutine(StopTalking());
            }
            else if (networkClient.Key == 1)
            {
                if (MicLoudness*100 <= 1f && endTalking)
                {
           
                    //FixMouthServerRpc();
                    return;
                }

                endTalking = false;

                mouth2.Value = networkClient.Value.PlayerObject.GetComponentInChildren<SkinnedMeshRenderer>()
                    .GetBlendShapeWeight(1);
                networkClient.Value.PlayerObject.GetComponentInChildren<SkinnedMeshRenderer>().SetBlendShapeWeight(1,
                    _skinnedMeshRenderer.GetBlendShapeWeight(1) + Time.deltaTime * reverseMouth);
                if (networkClient.Value.PlayerObject.GetComponentInChildren<SkinnedMeshRenderer>().GetBlendShapeWeight(1) <= 0f)
                {
                    reverseMouth = -1f * reverseMouth;
                    networkClient.Value.PlayerObject.GetComponentInChildren<SkinnedMeshRenderer>().SetBlendShapeWeight(1, 0.01f);
                }
       
                if (networkClient.Value.PlayerObject.GetComponentInChildren<SkinnedMeshRenderer>().GetBlendShapeWeight(1) >= 100f)
                {
                    reverseMouth = -1f * reverseMouth;
                    networkClient.Value.PlayerObject.GetComponentInChildren<SkinnedMeshRenderer>().SetBlendShapeWeight(1, 99f);
                }
                StartCoroutine(StopTalking());
            }
            else if (networkClient.Key == 2)
            {
                if (MicLoudness*100 <= 1f && endTalking)
                {
           
                    //FixMouthServerRpc();
                    return;
                }

                endTalking = false;
               
                mouth3.Value = networkClient.Value.PlayerObject.GetComponentInChildren<SkinnedMeshRenderer>()
                    .GetBlendShapeWeight(1);
                networkClient.Value.PlayerObject.GetComponentInChildren<SkinnedMeshRenderer>().SetBlendShapeWeight(1,
                    _skinnedMeshRenderer.GetBlendShapeWeight(1) + Time.deltaTime * reverseMouth);
                if (networkClient.Value.PlayerObject.GetComponentInChildren<SkinnedMeshRenderer>().GetBlendShapeWeight(1) <= 0f)
                {
                    reverseMouth = -1f * reverseMouth;
                    networkClient.Value.PlayerObject.GetComponentInChildren<SkinnedMeshRenderer>().SetBlendShapeWeight(1, 0.01f);
                }
       
                if (networkClient.Value.PlayerObject.GetComponentInChildren<SkinnedMeshRenderer>().GetBlendShapeWeight(1) >= 100f)
                {
                    reverseMouth = -1f * reverseMouth;
                    networkClient.Value.PlayerObject.GetComponentInChildren<SkinnedMeshRenderer>().SetBlendShapeWeight(1, 99f);
                }
                StartCoroutine(StopTalking());
            }
        }
    }
2

First off, doing multiple GetComponents per frame is horribly inefficient – much better to only do that once per player object and cache those. Second, you could drive the mouth movements through animations, in which case all you need to do is add a NetworkAnimator component (and make sure the audio is played at the same time).

But thirdly, if you’re going the route of sending specific operations over the network, don’t use a loop over connected clients, but instead use ServerRpcs and ClientRpcs, possibly with ClientRpcParams. The latter is an optional parameter to ClientRpcs that can make it target only specific clients.

Your code then becomes much simpler. On any client with a player that’s speaking, call a ServerRpc on the player object that calls a ClientRpc on the player object that replicates the mouth movement. This means you just need to pass the blend shape weights and anything else you might need to the ServerRpc, which then passes it on to the ClientRpc. Since you are calling these on the same player object (i.e. the player that is speaking is the same NetworkObject that other players should see speaking), there is no need to grab other player objects from clients.

Some uncompiled example code:

private void Awake() {
  mySkinnedMeshRenderer=gameObject.GetComponentInChildren<SkinnedMeshRenderer>();
}

public void UpdateMouth(bool endTalking,float mouthParam1,float mouthParam2) {
  if (endTalking) {
    //end talking here
   return;
  }
  mySkinnedMeshRenderer.SetBlendWeight(mouthParam1,mouthParam2);
  if (IsLocalPlayer) {
    if (!IsServer) {
      UpdateMouthServerRpc(endTalking,mouthParam1,mouthParam2);
    }
    else {
      UpdateMouthClientRpc(endTalking,mouthParam1,mouthParam2);
    }
  }
}

[ServerRpc]
private void UpdateMouthServerRpc(bool endTalking,float mouthParam1,float mouthParam2) {
  UpdateMouthClientRpc(endTalking,mouthParam1,mouthParam2);
}

[ClientRpc]
private void UpdateMouthClientRpc(bool endTalking,float mouthParam1,float mouthParam2) {
  if (IsLocalPlayer || IsServer) {
    return;
  }
  UpdateMouth(endTalking,mouthParam1,mouthParam2);
}
3

Thanks for the quick reply! I tried your code and its the closest I got it to working. Basically now the Host sees their own mouth move on both him and other players but the client sees the correct result. Did I do something wrong? I’m relatively new to netcode so this is all a bit confusing for me :slight_smile:

private void LateUpdate()
    {
        if (MicLoudness*100 >= 1f && !endTalking)
        {
            UpdateMouth();
           
        }
    }

 public void UpdateMouth() {
        _skinnedMeshRenderer.SetBlendShapeWeight(1,_skinnedMeshRenderer.GetBlendShapeWeight(1)+Time.deltaTime * reverseMouth);
        if (_skinnedMeshRenderer.GetBlendShapeWeight(1) <= 0f)
        {
            reverseMouth = -1f * reverseMouth;
            _skinnedMeshRenderer.SetBlendShapeWeight(1, 0.01f);
        }
       
        if (_skinnedMeshRenderer.GetBlendShapeWeight(1) >= 100f)
        {
            reverseMouth = -1f * reverseMouth;
            _skinnedMeshRenderer.SetBlendShapeWeight(1, 99f);
        }
        if (IsLocalPlayer) {
            if (!IsServer) {
                UpdateMouthServerRpc();
            }
            else {
                UpdateMouthClientRpc();
            }
        }
    }
    [ServerRpc]
    private void UpdateMouthServerRpc() {
        UpdateMouthClientRpc();
    }
    [ClientRpc]
    private void UpdateMouthClientRpc() {
        if (IsLocalPlayer || IsServer) {
            return;
        }
        UpdateMouth();
    }
4

I think the only thing missing is that UpdateMouth in LateUpdate should only be called on the local player (or, if you prefer, only on the server). The Rpcs then ensure the result is replicated for other clients. So do a check for IsLocalPlayer or IsServer depending in LateUpdate.

5

Still the same issue occurs. The client can see the correct movement but the host doesnt see movement in the clients mouth. I tried it with both IsServer and IsLocalPlayer both didnt work. IsServer makes the host see his mouth on both him and the clients and IsLocalPlayer does not show any movement on the client when you are host.

6

Ah, now I understand what you mean, my bad! Yes, if LateUpdate runs on a local client then the Server never calls UpdateMouth() itself. To fix this, change the ServerRpc:

    [ServerRpc]
    private void UpdateMouthServerRpc() {
        if (!IsLocalPlayer) UpdateMouth();
        UpdateMouthClientRpc();
    }
1 Like
7

You’d send one packet, it would be the ability for them to trigger the animation of the mouth, and the time stamp so they can project the animation Xmillisconds from when it started

1 Like
8

Thank you sir! This works perfectly.