Sample Texture from Cubemap Based on a Direction

Hi,
I’m trying to sample an area of a cubemap based on a direction. This is to get the average brightness behind the player from any direction.

Gathering the cubemap allows me to retrieve any one particular face of the cubemap. However, if my sample area overlaps multiple cubefaces this becomes a problem.

If I only needed the side faces, I could just stitch them together and sample a square. However, when I introduce the up and down faces that doesn’t work anymore. How would I best solve this?