yeah, that's how I first ran into the problem. Can be worked around by rotating the entire view but given that snapping also doesn't seem to work and there are other mask controls (like grouping and centering and so on) that are missing, that's a major pain
Looks like this was due to the way of each anchor points moving from first keyframe to the 2nd keyframe going straightforward, without seeing the shape.
In case you need a rotation of more than 180°, you just needn't get confused because there is no 360° setting at a first glance. The range is from -180° to +180°. So just start with a negative degree value.
Strange. Rotating a Bézier Mask works fine for me.
Which Vegas Pro version do you use? I could try sharing a project file of a working rotation.
Drawing a circle or rectangle is a matter of ratio dependend on your source's ratio. For HD adjust the width, then multiply that value with 1.7777 and you got the same height. This works for both the rectangle (to square) and oval (to circle).
No need to go into the math. I developed 3d visualization software for a living.
The problem isn't the math, it's trying to get evenly distributed bezier points without a grid to snap to. That's the other thing, snapping doesn't seem to work at all.