Game runs smoothly in landscape mode but lags/stutters in portrait!

Developed simple endless ball game. It runs smoothly in landscape(60 FPS) mode but stutters in portrait mode with 40 FPS very frequently(also this happens when game enters active mode from background). At this point when it runs at 40 FPS in portrait , when I turn to run in landscape, it runs smoothly at 60 FPS

Any idea what might be the cause the issue? Let me know if any specific details are needed

Have you tried to use Unity Profiler to see what is the difference between those modes?