Edge detection is accomplished via a first-derivative convolution kernel. In its simplest form it looks like [-1, 0, +1] in the one-dimensional case. If you prefer the normalized version, it is [-0.5, 0, +0.5]. Though this example is one-dimensional, it can be applied to two dimensions by merely taking the transpose of the kernel and applying it to the results of the first operation. Alternatively, you can just define a two-dimensional kernel matrix, which in this case is a [3x3] matrix, and apply it in a single step.

Before edges are to be detected, it is important to perform some level of Gaussian smoothing to eliminate pixel noise such that the edge gradients found by the convolution kernel are the "true" ones. This will make the step of non-maximum suppression easier. If the goal is to find a series of edges for segmentation, then one must also form edgels (i.e. edge-elements) from edge pixels and then perform edge-linking to create cyclic connections (your masks).

There are actually a few more steps involved, but it would take too long to post them here. However, if there is interest, I can post my lecture notes that I use for teaching this topic in machine vision.

Peter

"Edge detection is accomplished via a first-derivative convolution kernel."
That's a linear high pass filter as I surmised. Stay tuned.

Thanks Tech Diver. I also work in the field of machine vision, mainly PPT, Cognex and some Keyence. So I understand where you are going, but I don't see any way of controlling how the Bezier tool in Vegas interpolates the edge. Are you suggesting I apply filtering effects to the image prior to applying the Bezier mask?

I wasn't really suggesting any particular procedure in Vegas. I was just explaining how edge detection can be calculated. However, if you would elaborate a bit on what you are trying to achieve, we can probably make a few suggestions, etc.

Peter

[I]"How is edge detection accomplished? I find that images with sharp edges mask very well, but washed out, low light images not so good. "[/I]

Vegas doesn't do "edge detection", it only has the ability to manually create Bezier masks.

As you've rightly noted soft images are very hard to mask well and Vegas doesn't offer a whole lot of help apart from basic feathering.

Apps such as PS do offer more, PS now has a "refine edges" tool and has long had the ability to expand, shrink and smooth masks.
The Refine Edges tool seems to allow one to handle the problem of bleed at soft edges which is similar to what the more advanced chroma key FXs can do. If you don't mind spending a lot of time futzing around with multiple masks in PS you can do just about anything. There's plenty of good tutorial about doing this on YT.

Whenever I've needed to mask something difficult I use PS to wrangle it. Using the right tool for the job is the way forward. Vegas is no great shakes at masking and PS is dead in the water when it comes to mixing audio :)

Bob.