Can't figure out why my Client is slower than Hosts

The Problem
I can’t figure out why my client is much slower than the hosts, where the host has the correct speeds.

My logic
The client and client owner reads input, the input is then sent to a function where it calculates new positions based on Time.fixedDeltaTime and then those are sent to the server via ServerRpc. Afterwards, the client fetches the data and updates the transforms of the objects.

Thanks in advance.

    private void Update()
    {
        if (IsClient & IsOwner) ClientInput();
    }
    private void FixedUpdate()
    {
        if (IsClient & IsOwner) ClientToServerUpdate();
        ServerToClientUpdate();
    }
    private void ClientInput()
    {
        _mousePosition = Input.mousePosition;
        _forward = Input.GetKey(KeyCode.W) || Input.GetKey(KeyCode.UpArrow);
        _back = Input.GetKey(KeyCode.S) || Input.GetKey(KeyCode.DownArrow);
        _left = Input.GetKey(KeyCode.A) || Input.GetKey(KeyCode.LeftArrow);
        _right = Input.GetKey(KeyCode.D) || Input.GetKey(KeyCode.RightArrow);
    }
    private void ServerToClientUpdate()
    {
        _tfBase.position = ServerBasePosition.Value;
        _tfBase.rotation = ServerBaseRotation.Value;
        _tfTower.rotation = ServerTowerRotation.Value;
    }
    private void ClientToServerUpdate()
    {
        float baseRotationAngle = 0;
        Vector3 basePosition = _tfBase.position;
        Quaternion baseRotation = _tfBase.rotation;
        Quaternion towerRotation = _tfTower.rotation;

        if (_forward) basePosition = Vector3.MoveTowards(basePosition, basePosition + _tfBase.right, moveSpeed * Time.fixedDeltaTime);
        if (_back) basePosition = Vector3.MoveTowards(basePosition, basePosition - _tfBase.right, moveSpeed * Time.fixedDeltaTime);

        if (_left) baseRotationAngle = -90;
        if (_right) baseRotationAngle = 90;
        if (_left || _right)
        {
            Quaternion targetBaseRotation = Quaternion.Euler(new Vector3(0, baseRotationAngle, 0));
            baseRotation = Quaternion.RotateTowards(baseRotation, baseRotation * targetBaseRotation, baseRotationSpeed * Time.fixedDeltaTime);
        }

        if (Physics.Raycast(_camera.ScreenPointToRay(_mousePosition), out RaycastHit hit, float.MaxValue, layerMask))
        {
            Quaternion targetTowerRotation = Quaternion.LookRotation(hit.point - _tfBase.position);
            targetTowerRotation = Quaternion.Euler(new Vector3(0, targetTowerRotation.eulerAngles.y - 90, 0));
            towerRotation = Quaternion.RotateTowards(towerRotation, targetTowerRotation, towerRotationSpeed * Time.fixedDeltaTime);
        }

        if (_oldPosition != basePosition || _oldRotation != baseRotation || _oldTowerRotation != towerRotation)
        {
            _oldPosition = basePosition;
            _oldRotation = baseRotation;
            _oldTowerRotation = towerRotation;

            UpdateClientTransformServerRpc(basePosition, baseRotation, towerRotation);
        }
    }
  
    [ServerRpc]
    void UpdateClientTransformServerRpc(Vector3 position, Quaternion rotation, Quaternion towerRotation)
    {
        ServerBasePosition.Value = position;
        ServerBaseRotation.Value = rotation;
        ServerTowerRotation.Value = towerRotation;
    }

Client Print: ServerDeltaTime: 0.01594734 -- MoveSpeed: 1 -- BaseRotationSpeed: 50 -- TowerRotationSpeed: 65

Host Print: ServerDeltaTime: 0.01667547 -- MoveSpeed: 1 -- BaseRotationSpeed: 50 -- TowerRotationSpeed: 65

It seems too me, as if your client is already calculating the next movement before changing its own position. This is still done from the old position and only a bit later it really gets moved. This way your client calculates the same step multiple time and is therefor slower. The host is at the correct speed since it updates its position without delay.

You could safe an extra version of your position and rotations on the client and calculate according to these. This way you can use the current values.