While I am not sure what could be causing that, have you tried feathering the mask a little? If nothing else, that should help hide that line.

I have to take back what I said about it "not being transparent pixles in Mocha Pro". It is. When I import the image into my Mocha Pro plugin and I look at the image and only the image, it looks the same as in Gimp, but when I look at it as a "composite" or a "cutout" it changes the image. Increases the pixel resolution, and adds a transparency.

I have asked about this in the Mocho Pro forum on the BorisFX website. Although it would be ideal if the transparency did not create this problem in Vegas in the first place, as it shouldn't.

The transparent pixels show a BLACK background, rather than the lower layer on the Vegas compisition, as it should. That seems to be what is causing the problem.

This image highlights the problem a little better.

In the Mocha Pro plugin, everything looks fine, but after exiting the Mocha Pro plugin and coming back to Vegas to look at the composition (which has several layers) this artifact appears.

It seems that it has something to do with transparent pixels not having the correct "background". In other words, the transparent pixels don't "show the next layer behind" but instead show a black matte, or something like that, which is why they appear darker.

I don't know if this problem is from the Vegas Pro side or the Mocha Pro side, but everything looks fine in the Mocha Pro plugin before I exit out into the Vegas Pro composition.

Sounds like a premultiplication issue