Are you talking about a crude, non transparent white halo? Is it a 32 bit uncompressed file?

If so, did you try different options in alpha channel setting (in your event properties)?

What alpha channel settings are you using in Red? Use premultiplied- not straight.

I use Red3 and haven't seen the white halo you're referring to.
But then, I use Red outside Vegas. I use the standalone engine and bring my material into there, do my compositing and render out an uncompressed clip. I bring that clip back into Vegas.
I now own Serious Magic's Ultra plug, and am looking forward to seeing if keys are better in there than the ones I get from Red. Or if they're faster.
I have also found that keys in Vegas (I was using 5 at the time) can be darned ok, as long as they are lit well to begin with (an amber backlight, I'm told, creates a better separation between FG and BKG).
In Vegas, are you starting with a secondary colour correction to saturate the greens? Then the chorma blur then the chroma key. Shouldn't be any glow there (but there will be some smeering - is that the same thing?).
I just wish Boris had a secondary colour corrector like Vegas does. GREAT feature.
Still, I've heard from too many sourses that, "You will never get a great key out of DV(25)." I've come to believe it. I've come to think of it as an effect, not an illusion.

Doing some tests...

How can I tell if the file is 32bit uncompressed?

I capture via firewire into V6 using a Sony DSR-11 DVCAM deck.

We also put the phootage through Serious Magic's Ultra. It does an awesome job at Chroma Keying... just can't figure out how to add the Overlay/Cutout that also needs to be there. I'm a firm believer in becoming a Master of 1 program (provided it can do what you need) as opposed to knowing a little about a lot of different programs. Especially with the cost of the individual programs.

I'm pretty sure we have what we need to make it happen w/ Vegas6 & Boris RED 3GL. They are both VERY powerful so I'm sure the problem is with the operator (me) and not the programs. Currently we have satisfactory results but I can still see where I can improve. Maybe I'm just being too hard on my self... but that's the only way to get better, ya know. Plus, I like to correct things before anyone else has a chance to notice them. It's the worst trying to 'back your way out of a corner' by explaining why it's not as good as it can be.

I agree. And I tried for well over a week trying to get a perfect key out of both Vegas 5 and Boris RED3GL. I sent examples to this forum, the Cow and directly to the folks at Boris. I never got the perfection I wanted. It just ain't the $10+K setup that can make perfect keys from 4:4:4 sources. For my meager needs, I start with seeing how the key works on Vegas. If I'm unhappy with that, I go to Boris.
But either way, as I said above, I now consider it to be an effect. It's never going to fool anybody.
Still, I've seen some EXCELLENT stuff on the Ultra demo. I just need to sit down, clear some hd space, install that jewel and learn it!
A GREAT piece of advice they were GIVING AWAY at NAB is dynamite! When you want to shoot your subject, turn your camera sideways (assuming you're shooting a person). Now you have an image that cam be head to foot, and since your (non-HDV) image is now 480 wide by *720* high!, you have a lot more pixels to play with. You can push in to a mid shot without loosing any resolution! Brilliant!

The "White halo" - effect only appears if you use Boris Red as a plugin in Vegas. If you render the effect from Red and import the clip instead, it looks great. It's really wierd.

But you can set the Level-slider on the video-track to 99.9% instead of 100%, then most of the halo dissapears if you use Red as an effect.