Collisions triggering sporadically at best

I have two objects colliding (bullets and enemies), and I have it set so that the bullets are destroyed when they collide with the enemies.

However, in actuality most just pass through the enemy without triggering a collision. Only a few actually trigger a collision.

Here is a video demonstrating it (the enemies are destroyed once they have been hit twice)

I am quite new to Unity so any help would be appreciated.

Are you setting your Collision Detection correctly for your rigid bodies? http://unity3d.com/support/documentation/Components/class-Rigidbody.html