Yeah, it's a pain isn't it. When you get to the point where the edge artifacts are going away, you start losing parts of the image too.

My way around it was to buy Adobe Premier Elements just to use for it's vector keying. Adobe started putting this keying technology into the latest versions of their software after they acquired Serious Magic.

Cinegobs looks really good as well but it doesn't seem to work on my system.