If you spherized the whole frame and then cut a piece out of that what would be different if you cut a piece out of a frame and spherized the result?
Answer, none.
However as the newsprint FX affects the levels you will see the FX being applied to the black part of the frame after the cutout is applied.
I think the cookie cutter works differently because it is an FX not a composite. In the case of the CC the result relies upon the order of FX in the chain.
Hope I've explained this correctly, took me a while to get my head around it. And I agree that little yellow triangle never grabbed my attention until someone mentioned it.
However as the newsprint FX affects the levels you will see the FX being applied to the black part of the frame after the cutout is applied.
shouldn't the cutout tell Vegas I don't want to see the black part of the frame. Regardless of how the FX affects the levels of the black area, I have already deliberately set up a mask to indicate what I want to see.
I don't think it specifically has to do with the bezier mask -- it's the newsprint FX itself. That's just how it affects transparent areas. Apply it to a text event that has a transparent background, and you'll get the same result. Ditto for, say, a still image that makes use of transparency.