I've searched the new VMS4 documentation, including the PDF manual for any description of Picture-in-Picture features, and cannot find any. Is this feature still in VMS4?
You can use Pan/Crop for this. Put the background video on the video track and the inset video on the overlay track. Stretch the crop area larger than the frame size and the image will be reduced. Place the PIP in the frame by sliding the cropping rectangle around in the opposite direction.
Cookie Cutter is there in the Video FX, but as I look at it, it only allows a small portion of the overlay video to be shown. Using the Pan technique allows the whole picture in the PIP.
Of course I am new so I just may not know how to properly use the cookie cutter PIP.:-)
You can use both techniques together to get the whole picture in a circular 'frame' with a feathered edge :-) . Example - my wife took some video of me and our 2 sons in one of those wild west pictures with cutouts where you poke your face through. I was able to take some footage of my wife and arrange things so that it looked like she was in the picture as well. It wasn't completely convincing, but it was fun!
A lot of MS' power comes from being able to combine things like this.