HoloLens Input Module Inconsistent

The Input Module is acting very inconsistent, I have a simple button in the scene and it's not being Highlighted, when the cursor is on it, but for example on the cursor is far off the button. The raycasts for the UI are somehow very off, but I don't understand what is the reason behind it. Any ideas?

Microsoft has been working on a replacement. Try it out here: https://github.com/Microsoft/HoloToolkit-Unity/pull/319