Physics interaction using mouse movement

Hi there! I’m working on my own code for a physics based interaction system, like doors, drawers, levers, pickups, buttons etc.

All good, the problem lies that the force that’s been applied to the physics object is dependent on the camera vectors, so if i’m straight over a physics object or straight above, and i’m trying to move the mouse forward or backwards, the vector is been applied up or down, which makes sense because it’s using the cameras vector. Now the question: How could i implement a calculation that the force that is applied to the physics objects are ALWAYS applied dependent on the movement of the mouse input so it feels right?

Video demonstration:

The code for the Direction:

public virtual Vector3 GetDirectionGrabForce()
{
    Vector3 forceDirection = Vector3.zero;
    if (IsInteractorInteracting())
    {
        Vector2 lookInput = PlayerInput.Instance.GetLookInput();
        Transform camTransform = GetInteractor().GetCamera().transform;
        forceDirection = camTransform.forward * lookInput.y + camTransform.right * lookInput.x;
        forceDirection.Normalize();
    }
    return forceDirection;
}

An this is how i try to convert the world direction to the local Direction so i can apply the force to the correct axis:

currentDirectionInput = GetDirectionGrabForce();
Vector3 local = transform.InverseTransformDirection(currentDirectionInput) * -1;
local.Normalize();

linear += local.y * Time.deltaTime * forceMultiplier;

I hope you understand what i want to achieve.
Thanks!