It also depends on the colors you use and the codec you use for output, e.g. this may happen due to 4:2:0 color reduction.

Are you working with interlaced footage?

It's a 1080p 4.2.2. 8bit file project from my Atomos Shogun Flame, in DNxHD. All progressive. I see it in the rendered file too. Snapshot and render settings are best/full.

So...any ideas, hints, thoughts?

So...any ideas, hints, thoughts?

I use mask tool for this.

You need AA on your circle. A slight blur should do the trick.

Thanks to all, but..the mask circle is just as jagged, and it can't do borders. don't really want to use the blur, as I need them to be very crisp. I find it strange that Vegas, despite of all its complex capabilities, cannot draw a smooth circle, but makes it look like some 8 bit graphic from 1995.

Yes, Vegas can do this. It has nothing to do with 8 bit or 32 bit, you just need enough resolution. Make sure your project is high enough resolution, then add a little bit of feather (say 0.010), then preview at best/full

If that still isn't good enough for your purposes, then import a .png circle with transparency made in photoshop, etc. Same principles apply though, you need a little bit of feather/blur to make it look natural.

Thanks Kinvermark,

my project resolution is as high as can be. 1080p all set to full/best.

The photoshop circled do not look any better for that matter, which I really cannot quite understand in this day and age of high end compositing technology. Even the slightest blurs don't look good on the circles unfortunately. It gets soft immediately.