I would definitely advise against using your quaternion rotation’s z-component in the reflection calculation. Generally, unless you’re doing some very esoteric stuff, you don’t need to use quaternions’ component values as the maths if very opaque and hard to visualise.
Just to back this up, Unity’s documentation says something similar under the Quaternions header (along with a bit of extra info) : Unity - Manual: Rotation and Orientation in Unity
To reflect your bullet’s direction, we need to find the normal vector of the hit surface - the direction vector pointing out from the hit surface like a flag pole, perpendicular to the surface (https://bit.ly/3NlbJ9c).
If you were colliding with a mesh I’d say you could raycast forward and then if you hit something the hit object would give you the normal vector automatically however it appears that here your boundary is a circle defined by your BorderRadius.
This means we can take advantage of the fact that the vector from the centre of the circle to the position of the collision is in the exact opposite direction to the hit surface normal vector. I’ll continue with the approximation that your bullet’s out of bounds position as the collision position from which to determine the hit surface normal as it’ll assist in understanding the overall process but if you want to be more precise about the reflection then you’ll need to calculate where your movement vector intersects with the bounds circle.
Once you’ve determined that your bullet it out of bounds then the vector from the centre of the bounds to the out of bounds position is ‘bullet position - bounds origin’ but since it appears your bounds origin is at 0,0,0 then it’s just bullet position (which is transform.position in the bullet’s script). Therefore the hit normal is this vector multiplied by -1.
Now that we know the hit surface normal vector we can use Vector2.Reflect to get the reflected direction:
Vector2 hitNormal = -transform.position.normalized;
Vector2 currentDirection = transform.up;
Vector2 reflected = Vector2.Reflect(currentDirection, hitNormal);
transform.rotation = Quaternion.FromToRotation(transformUp, reflected) * transform.rotation;
So in this code we get the hit surface normal using our approximation, use the transform.up vector as the movement direction (which we can do since you’re using the translate function passing only a speed to the up/y component), reflect this vector across the hit surface normal to find our mirrored direction vector then rotate our bullet from its current direction to the reflected direction.
This is how quaternions are typically used, they’re created and modified using the static functions within the Quaternion class. This nicely hides all the maths that we don’t need to deal with for these sorts of features. The only exception that you’ll notice is that I multiply two quaternions on the last line - a quaternion can represent an orientation or a rotation, this multiplication when we want to apply a rotation to an orientation. If you feel like there’s a lot to take in here that’s because there is but I really believe this sort of maths is absolutely indispensable for game programming so please bare with it!
The only other thing in your code that will trip you up is your out of bounds test. You currently use
this.transform.position.x < -BorderConstants.BorderRadius which will only check if it is out of bounds on the x-axis when moving left in the negative direction. For a circle we should instead use the check
this.transform.position.magnitude > BorderConstants.BorderRadius as this will check whether the distance from the bounds centre to the bullet is greater than the radius you’ve defined. Note, you don’t need to use ‘this.’ unless you’ve defined some other variable within your function called ‘transform’ and in that case I’d advise you rename that variable instead of use ‘this.’.
Let me know how you get on with this, if I’ve made any bad assumptions or if you have any questions.