The Quat vs Euler thing I think is the lesser problem here (and personally I’d just do Quat, because I almost always use Quat… I only use euler’s when I know a staticly defined orientation, since they’re human readable).
The problem is the ‘viewport’. So we need to consider the view frustum, which describes the entire 3d space that the camera can actually see.
The view frustum usually has the shape of a pyramid with its head cut off (though other shapes can be found with different projections, such as a cuboid shape for orthographic projection). It looks a bit like this:
Note, the frustum if it had its pyramid head, the pinnacle of the pyramid would be located at the camera lens. Where it’s chopped off is defined by the ‘near’ plane.
The problem that arises is where is the cannon point of origin located (where does the projectile begin)?
If it exists at the same position as the camera, then we know that we can rotate left or right from facing forward one half the horizontal field of view.
BUT, if the cannon point of origin is located somewhere other than the same position of the camera, well… how far you can turn depends on the range of the projectile.
If the cannon is in front of the camera, and it’s range is very short… it can turn very far to the right or left since it wouldn’t exit the view port. Where as if its range is very far, you wouldn’t be able to turn very much, else you’d strike the viewport edge way far out at some point. This also goes for the vertical, as well as angle of view relative to the camera. If the camera is slightly above the cannon and looking down at it, then you see much more below the cannon, but not above the cannon.
All of this complicates the calculation of it a bit more than the prior.
BUT, it all hinges on that view frustum.
OF COURSE
The easier method would be to just get a ‘best guess’. If the viewport and cannon stay in similar relative positions, just play with the numbers until you find what feels like a good max turning angle… damned if it slightly overshoots or undershoots the viewport edge. No complex view frustum math needed.
Then once you have that max change. You merely just compare the angle between the shot direction and the camera foward to make sure it doesn’t exceed this range.