STM32 Custom HID 6DOF tracker

I have managed to get a STM32 with custom HID implementation to send absolute tracking position out, and are using it to track an object.
However when i compare it to a Vive tracker 3.0 i experience some latency around ~50 ms of latency which is to much for my application. As i have an additional 25 ms in the STM32 it self.
I have tested the STM with a python script and the Vive together and I only have 1.5 sample of latency, which is 15 ms between them. I know that the overall latency of the STM is around 30 ms, from starting to move to have received data on the PC through USB, and send data back out the USB to an USB analyzer to be no more than 30 ms. So i know that the problem is not with my STM device.

I tried to set it to an FixedUpdate and 0.001 update rate, does not improve it.
My question then comes, is there any way to adjust a custom device to have as low latency as possible, such as if i were using a 1000 Hz mouse or a Vive tracker? Is there any settings im missing for getting such a device to run as fast?