Optimization of bullet hit calculation per frame

My bullet objects travel during several seconds.

I was originally calculating the hit point and generating a Bounds object so I could easilly tell when the bullet had impacted on the surface. This accounts for the bullet final impact point.

However, I need to calculate whether a bullet impacts on a game entity every frame. For this I am now casting a ray every frame from the curret position to “newPosition”:

// Check if the bullet hit a player this frame 
var hitInfo : RaycastHit;
if (Physics.Raycast(transform.position, bulletSpeed.normalized, hitInfo, (newPosition-transform.position).magnitude, 1<<8 )) {
    if (hitInfo.collider.tag == "Player") {
        // The bullet hit a player entity
    }
}

Is there any way I can optimize this (perhaps trying a sphere collision before casting the ray), or is a “raycast per bullet per frame” the quickest I can come up with?

(My game uses physics but I am not using Rigidbody for bullets as that is noticeably slow).

I ended up doing:

If gravity is applied, I cast 1 short raycast per bullet per FixedUpdate.

If no gravity is applied, I precalculate the final hit point but still cast a ray per bullet per frame, in this case including only the “players” layer: