When recording I’m using the Apple Intermediate Codec - after that i convert it to H264 in QuickTime.
I have both, the HD and non HD version (for 10.4) but i prefer the non HD version which works fine on 10.5.8.
Maybe you should try to set the recording frame rate higher. I always use 30 fps in iShowU and it looks really good.
But anyway i guess the problem is that you never can synch the frames of the source with the recording software and some content works better than the other. I sometimes quickly recorded parts of DVDs with iShowU and had drop frames and slightly stuttering when the camera moved.
Using a camera to record it will also record the latency of the displays pixels and the latency of the cameras chip but as it will never record at a higher frequency than 25 or 29.97 fps the chance of getting 2 frames in one is exactly the same it will be just smoothed by the latencies.
iShowU HD is was not working for me (it captures blurry/off center pixels), but the regular version of iShowU works better. In either version, it’s very difficult to position the Unity window capture area.