Smart Mask (AI assisted automatic masking) Introduction

fr0sty wrote on 1/23/2023, 12:03 PM

This is an incomplete feature that is being released in Beta form. We need your feedback to continue its development.

Here's his introduction written by VEGASPascal for the new smart mask feature. Please check it out, let us know what works and what doesn't for you! New updates will continue to be added to improve this functionality as it comes out of beta in the future.

Hi Community,

This is a very early drop of the "Smart Mask" feature. The status of the plugin is "work in progress" but we want you to give a first impression of the feature.

Here some quick notes:

this is only an overlay plugin... nothing happens in the render process (yet)

the goal is to create a mask the user can send to the Bezièr plugin

AiA plugin (artificial intelligence assistant): user is still able to improve bad Ai results in the edit mode of Bezièr

it is a CPU only plugin (GPU precision was too bad)

new overlay function of VEGAS (OFX draw suite)

known bug: After the first time you press the "Analyse" button nothing happens in the preview. You have to restart VEGAS and start again.

Info for the plugin

Analyse: Starts the analyse process (object detection) for the current frame

ShowLabels: Shows the confidence score and the class label for each object. (only visible after analyse process)

Confidence: Filter for objects with a given confidence score (only visible after analyse process)

Shape: Improve the contour of the selected shape (only visible after selecting an object)

Transfer: Send selected shape to Bezièr plugin (only visible after selecting an object)

Boolean parameter in timeline shows the analysed frame (only visible after analyse process)

We hope you enjoy the plugin.

Back to post

Comments

Howard-Vigorita wrote on 2/5/2023, 9:15 PM

you were mentioning shooting infront of a uniform color to help key, did you do that here, or is the AI actually seeing a black background? In my experience black backgrounds don't work well if your subject has areas of low contrast on the boundaries (which makes sense) but I also see why someone would try with a black background, flaws in tracking not so obvious. I've given up on black/dark backgrounds so many times when using AI masks due to strobing issues. Try a well lit background and foreground. Good to see your work flow and mini tutorial

@Todd-A0 In most of their stage show there's usually lighting effects in the background but they chose to do this little brass diversion against an unlit backdrop. You're right though. Knocking out blacks often doesn't work with shadows and black mic stands all around. But it might have worked here to isolate all the performers since I knocked out to the original scene on the track underneath. But to knock out everything but one performer could only be done with a custom mask. Could also turn the group into a musical analog of Fantastic Four. By isolating each performer separately on their own track and applying a different special effect to each one.

Mindmatter wrote on 2/6/2023, 4:42 AM

FWIW, I've had pretty good results using a huge white background lighted to 100%, which then served as a luma key. It was on a project with many kids on a 6m stage where green screen was not an option. It worked so well that I started to wonder if it wasn't a better allround solution than green screen. It has its own set of issues though.

The only critical part was when the light on the kids in some scenes was not super optimal or when their clothing was too bright, in which case ther would be some transparency issues. The shadows they cast on the floor were also a problem in certain scenes.
This was the production, all keying was done with a luma key in the white background.

paul-marshall wrote on 2/7/2023, 4:59 AM

You're better off using VEGAS Motion Tracking to do that, it's better designed for motion tracking non-huma/animal objects. Tutorial here:

Thanks for the tutorial (all the tutorials) - thats how I do it at present, but you asked for thoughts on the smart mask and it has the potential to be easier with less clicks and an easier workflow than via Motion Tracking, which still needs to make a mask. I like the way the AI creates a mask that matches the shape pretty well. So then select Bezier Mask/General options and motion track it. Select Mask FX. Add Blur effect and move blur to before the Mask and it's done!
In my brief play with it, it found many different inanimate objects so I dont see why signage could not be one of them!

Roger Bansemer wrote on 2/8/2023, 8:55 AM

I've watched the posts and tutorials here on "smart mask" but they all really seem to be about other masking procedures and not the smart mask. When I use it I see a rectangular outlines around objects and I can then create a very rough mask on that object. Where do you go from here? I'm hoping someone will post a short tutorial strictly about how to use the "smart mask". Thanks

Howard-Vigorita wrote on 2/8/2023, 11:27 AM

The update notes say Smart Mask is unfinished. Making the outline and transferring to Bezier is all there is right now. From there you use Bezier as before.

fr0sty wrote on 2/24/2023, 10:14 PM

Mindmatter wrote on 2/2/2023, 9:09 PM

If I understand this feature correctly, it should be able to replace green screen keying at some point, right? I suppose it should be pretty easy when the subject is clearly distinct from a flat background.

Unfortunately based on the Resolve experience, the answer is No. it is not a replacement for a green screen. The reason is you can perfect a single frame, but when the tracking occurs you can get a different result on every frame, assuming a moving subject. It just depends if you would not be embarrassed by a Zoom meeting type background replacement in the video you were producing. Repeatedly with Resolve I've had to ditch their AI masking because if it's not perfect it looks amateurish.

Color grading is a better task, say you wanted to alter exposure on your subject but not affect your background, or to separate your subject from a background maybe you reduce exposure or blur the background. Badly tracked frames aren't' as obvious when it's not a background/foreground replacement but a color grade instead. @Mindmatter

This is especially true if you slightly feather the edges of your mask, further hiding any imperfections.