UI Toolkit MouseManipulator class breaks on both middle and right mouse button when using ManipulatorActivationFilter

In Unity 6 preview 21 I have a bug when I use a custom class that inherits from MouseManipulator.
The Input System package version I have is 1.11.1 just in case it helps add information.

In my class I add to the activators list with a new ManipulatorActivationFilter that has one mouse button and the EventModifier value is set to None for the enum value.

public MouseRectDragManipulator(MouseButton mouseButton, EventModifiers eventModifiers = EventModifiers.None)
{
  activators.Add(new ManipulatorActivationFilter()
  {
      button = mouseButton,
      modifiers = eventModifiers
  });
}

// This works perfectly fine using MouseButton.LeftMouse
private MouseRectDragManipulator _panningManipulator = new MouseRectDragManipulator(MouseButton.LeftMouse, EventModifiers.None);


// Setting to MouseButton.RightMouse or MouseButton.MiddleMouse like below does not activate the MouseManipulator event handling.

private MouseRectDragManipulator _panningManipulator = new MouseRectDragManipulator(MouseButton.RightMouse, EventModifiers.None);

When I set the ManipulatorActivationFilter to use MouseButton.Left is works perfectly fine, but if I try to use MouseButton.RightMouse or MouseButton.MiddleMouse it breaks.

Has anyone run into this bug or does anyone know what might cause it.