If this isn't user error, it seem to be an incorrectly designed masking algorithm.

Here are 3 images of 2 layers with different masks enabled. In the bottom image, the area around the red oval should be filled with the color of the layer underneath. At no point in the process is black assigned to those pixels.

Vegas FX 3.011716.14403
Fast Computer
Win10 Pro

Thx