I have some footage where the background color is not solid enough (it's basically two tones of green - bad lighting) and I'd like to see if I can't change the background color and then do my colorkeying with the new color. Is that possible?
Perhaps someone else will have a Vegas trick that can do that well, but my past attempts in that direction were akin to painting myself into a corner. If you're actually seeing color remaining (not just a dirty alpha channel) after the key is applied, try putting a second keyer in the fx chain to remove the remaining color. That's a typical workflow in other non-Vegas keyers such as Keylight even for alpha channel cleanup.
A worst-case suggestion would be to duplicate your clip, apply secondary color correction to the copy to see if you can even out the background - not worrying about how your foreground looks. If you can get a decent key, you can always use the alpha channel from that key as a mask to allow the foreground of your original clip to show. If your source footage is anything highly compressed, the key edges may not be clean enough to use after processing, but give it a try. You may have better luck than I.
And depending on what your foreground consists of, you can try the Channel Blend FX in the chain after your keyer. I have been able to clean up alpha channels with that on occasion. Applying a small negative number in the green channel of the Alpha computation may clean up your key. You'll probably have to adjust both the keyer and Channel Blend to get the best results. It's not an intuitive plugin, but can do a lot to help an Alpha channel out.
"Perhaps someone else will have a Vegas trick that can do that well, but my past attempts in that direction were akin to painting myself into a corner."
There is no magic solution, all the tools that you could attempt to use face much the same challenge, differentiating one color from another. The Secondary CC FX is just the same as a chroma keyer with some extra bits added. In other words it first has to do what the CC'er does and from that selected region adjust the color, so Yes, you always end up painting yourself into a corner.
The best approach is to really look at the problem to see how much of it can be solved by the software with the least amount of human effort. The area that's most important is the area around the subject, the rest of the background can easily be removed with a garbage matte. Of course the area being manually masked out may need some keyframing but compared to rotoscoping the subject out the task is trivial.
Even the area around the subject can be dealt with by splitting it up. Maybe one setting works best for the hair, one for the face and one for the body. So split the problem up using masks and composite it all back together again. This takes a bit of work but again nothing like what rotoscoping frame by fame at the pixel level is going to take.
Always remember if you want Hollywood class results it helps to have their production values but even then they can spend a man week keying a couple of seconds. Be prepared to invest time on these problems.
In this case, there is no such thing as time. It's a thrice weekly show that lasts 5 or 6 minutes. It's normally a very easy job. But we've got someone with a horrible background this time. He has a greenish grayish background, a shadow and a graying beard and hair. Shit!
Amen to that.
Given what you are doing have you considered a hardware keyer?
WIth that you get to instantly see if it's working or not and given your time constraints I think the unit would pay for itself pretty quickly.
@Farss:
Any suggestions on a hardware keyer that handles HD? For all the time I spend struggling with software, hardware just might pay for itself in a jif.
"Any suggestions on a hardware keyer that handles HD?"
Not really, I've seen the Datavideo one in use but it is only SD, works very well.
Once you start talking HD video processing of any kind you do need deep pockets.
I just found the MBP-100CK Chroma Keyer that has HD-SDI I/O but (cough, cough) it is $16K from B&H.
You could also look at Black Magics Decklink Studio product, it does have a RT keyer but only works in SD, darn.