Android touches cause low FPS - what am I missing?

I’m seeing a very strange and disturbing performance issue on my android device.
When I do nothing and just look at my screen, it works smoothly (I have a character performing an ideal animation).
However, when I put my finger on the device (and hold it) it suddenly drops in FPS and looks terrible.
This is not related to my code, I checked and this happens without me even checking for Input.

I couldn’t find an answer in other threads, in my case I’m really not doing anything and touching the screen makes it lag. There is probably some project setting I’m missing, but I have no idea what.

To sum it up: Holding tap on android device causes huge lag (3D game, 2D was fine)

Could someone help me?

UPDATE:
I’ve ran some more tests, to further narrow the issues down:
I ran the game without the Canvas, without the EventSystem, without my scripts, just a camera showing 2 static objects and an animated one.

The issue reproduces. The animation is smooth until I touch the screen, at which point the FPS drops terribly. What can I do about this? Surely this isn’t normal.

This is driving me crazy, and I found this issue in a couple other threads, but it wasn’t answered anywhere.