At the bottom of the Pan / Crop window, next to the timeline, is a checkbox and a label that says Mask. Checking that enables bezier masking, which allows you to define an irregularly shaped mask of the event. Check the manual / online help for more info on using it.
Supposedly, you should be able to make a square shaped bezier mask and then adjust its tagents to form a circle. That said, though, I'd think you'd be better off using an FX to achieve what you want. Chromakey if you have a solid background or cookie cutter if you don't.
Another thing about the cookie cutter than can cause some confusion (well, at least it confuses the heck out of me still) is that the default size value for the shape is most of the frame. When i first apply the effect it seems like nothing has happened. If you get confused by this too then one of the first things to do is to make the size value smaller, then you'll see the effect happening.
And I'll throw in my tip-of-the-day for cookie cutter:
Temporarily enable a very thin border (red or yellow or something easy to see), at least while you are sizing and placing the cookie cutter. You can get rid of the border when you're done, but it makes initial placement and sizing of the shape a lot easier.