Rather than go back and forth with questions about your footage and lighting and settings, if you post a single image from your blue screen footage (unprocessed and in zipped TGA or high-quality JPG format, or render five frames of your unprocessed bluescreen footage as a DV AVI and zip it) to a website somewhere, I'd be happy to look at it in Vegas and send you my project so that you could use the same settings in your project.

Or email it to me at sonicfoundry at mavart dot com.

Thanks for the comments. I tried using a Gaussian blur, and it helps in some cases, but doesn't really address the real problem. It looks like the effect is in the mask generator in the Secondary Color Corrector. When it scans for colors, the processing seems to be done with large pixels. The result is that subject borders have strong pixellation around them. Increasing Smooth, or using a Gaussian filter, tend to make the subject edges partially transparent. For example, with Smooth set high, the subject edges become too indistinct.

Interestingly, the pixellation looks like it's from a vertically-applied algorithm. It's similar to using just vertical filtering on the Gaussian blur, but with wide pixels.

Any other ideas?

Is it DV footage? I remember running into problems with blue screen and DV compression. It sounds similar to what you are talking about, where you see evidence of vertical compression near the edges. And there is a difference between NTSC DV and PAL DV in this area so one might be better than the other for blue screening. But I'm hoping someone who knows what I'm talking about will jump in because that's all I remember.

Yes, this is NTSC DV footage, and it seems I've run across comments about the 4:1:1 color sampling causing problems. However, I tried the same techniques with both a blue and green background, and the effect seems similar in either case. That, and the size of the blockiness, had led me to believe that DV wasn't the problem. Maybe that's wrong?