Stabilization is very impressive, but it leaves some kind of unpleasant artifacts. Mainly I see a kind of "swimming" or localized moving distortion patches in stabilized clips.
There also doesn't seem to be much difference between the different stabilization settings.
I did quite a bit of experimenting with this feature on footage shot with a small camcorder at Yellowstone Park. The camcorder's image stabilisation wasn't very good so I tried to make the clips affected by shake suitable for viewing. Like you, I found that if you tried to stabilise a very shaky clip the result was a shimmering effect.
I settled with the general rules that, if the shake was only minor, use the 'Hand held camera, fast-moving scene' (irrespective of whether the scene was fast moving or not). This will smooth out the shake nicely but may introduce minor destortion. You make a judgement yourself if this is acceptable.
If the shake was moderate (you have to experiment a bit to determine what 'moderate' is), use the 'Hand held camera, slow-moving scene'. This will smooth the shake out somewhat but minor jerks will still be evident.
If the shake is severe, don't bother to use the stabilisation feature. The clip will be just as unwatchable.
One other thing. The first rule will smooth out nicely a pan that isn't very smooth, with very little noticable distortion.
The bottom line is, the stabilisation feature does the best job it can on a shaky clip but there is no substitute for a camcorder with a good optical image stabilisation system.
I'll play some more with different modes, but in general I think your conclusion is exactly correct.
I'm using a micro four third camera with an old Olympus OM 35mm still camera 50mm f1.8 lens for low light video so IS isn't an option. OTOH, perhaps a walking stick/monopod is : }
What appears to help a lot is upping the shutter speed. I normally shoot in 24P so the shutter is normally at 1/50th, but if I up it to say 1/80th or 1/100th the stabilization works quite a bit better. I think what's going on is the software needs to correlate spots in successive frames and getting rid of some of the blur helps.