Got me going again and I'm certain someone else was complaining about this as well so I thought initially "Argh right, that's the bug they were complaing about". Well I / they was wrong.
If you comp two almost identical things and add a feathered bezier to the upper track and then apply an FX to the upper track like Curves you get an outline where the mask is. Change the order with The Triangle in the FX, problem is banished.
Thinking it though, Vegas is probably doing exactly what I've told it to do, just that's not the outcome I expected. The Curve is altering the alpha data in the feathered section of the mask, result seems to be to make it lighter hence the outline. Changing the pre/post triangle in the FX means the transparency is calculated after the Curves FX has done it's thing, no level shift, no outline.
Bob.