Smart Mask (AI assisted automatic masking) Introduction

fr0sty wrote on 1/23/2023, 12:03 PM

This is an incomplete feature that is being released in Beta form. We need your feedback to continue its development.

Here's his introduction written by VEGASPascal for the new smart mask feature. Please check it out, let us know what works and what doesn't for you! New updates will continue to be added to improve this functionality as it comes out of beta in the future.

Hi Community,

This is a very early drop of the "Smart Mask" feature. The status of the plugin is "work in progress" but we want you to give a first impression of the feature.

Here some quick notes:

this is only an overlay plugin... nothing happens in the render process (yet)

the goal is to create a mask the user can send to the Bezièr plugin

AiA plugin (artificial intelligence assistant): user is still able to improve bad Ai results in the edit mode of Bezièr

it is a CPU only plugin (GPU precision was too bad)

new overlay function of VEGAS (OFX draw suite)

known bug: After the first time you press the "Analyse" button nothing happens in the preview. You have to restart VEGAS and start again.

Info for the plugin

Analyse: Starts the analyse process (object detection) for the current frame

ShowLabels: Shows the confidence score and the class label for each object. (only visible after analyse process)

Confidence: Filter for objects with a given confidence score (only visible after analyse process)

Shape: Improve the contour of the selected shape (only visible after selecting an object)

Transfer: Send selected shape to Bezièr plugin (only visible after selecting an object)

Boolean parameter in timeline shows the analysed frame (only visible after analyse process)

We hope you enjoy the plugin.

Back to post

Comments

Howard-Vigorita wrote on 2/5/2023, 9:15 PM

you were mentioning shooting infront of a uniform color to help key, did you do that here, or is the AI actually seeing a black background? In my experience black backgrounds don't work well if your subject has areas of low contrast on the boundaries (which makes sense) but I also see why someone would try with a black background, flaws in tracking not so obvious. I've given up on black/dark backgrounds so many times when using AI masks due to strobing issues. Try a well lit background and foreground. Good to see your work flow and mini tutorial

@Former user In most of their stage show there's usually lighting effects in the background but they chose to do this little brass diversion against an unlit backdrop. You're right though. Knocking out blacks often doesn't work with shadows and black mic stands all around. But it might have worked here to isolate all the performers since I knocked out to the original scene on the track underneath. But to knock out everything but one performer could only be done with a custom mask. Could also turn the group into a musical analog of Fantastic Four. By isolating each performer separately on their own track and applying a different special effect to each one.

Mindmatter wrote on 2/6/2023, 4:42 AM

FWIW, I've had pretty good results using a huge white background lighted to 100%, which then served as a luma key. It was on a project with many kids on a 6m stage where green screen was not an option. It worked so well that I started to wonder if it wasn't a better allround solution than green screen. It has its own set of issues though.

The only critical part was when the light on the kids in some scenes was not super optimal or when their clothing was too bright, in which case ther would be some transparency issues. The shadows they cast on the floor were also a problem in certain scenes.
This was the production, all keying was done with a luma key in the white background.

paul-marshall wrote on 2/7/2023, 4:59 AM

You're better off using VEGAS Motion Tracking to do that, it's better designed for motion tracking non-huma/animal objects. Tutorial here:

Thanks for the tutorial (all the tutorials) - thats how I do it at present, but you asked for thoughts on the smart mask and it has the potential to be easier with less clicks and an easier workflow than via Motion Tracking, which still needs to make a mask. I like the way the AI creates a mask that matches the shape pretty well. So then select Bezier Mask/General options and motion track it. Select Mask FX. Add Blur effect and move blur to before the Mask and it's done!
In my brief play with it, it found many different inanimate objects so I dont see why signage could not be one of them!

Roger Bansemer wrote on 2/8/2023, 8:55 AM

I've watched the posts and tutorials here on "smart mask" but they all really seem to be about other masking procedures and not the smart mask. When I use it I see a rectangular outlines around objects and I can then create a very rough mask on that object. Where do you go from here? I'm hoping someone will post a short tutorial strictly about how to use the "smart mask". Thanks

Howard-Vigorita wrote on 2/8/2023, 11:27 AM

The update notes say Smart Mask is unfinished. Making the outline and transferring to Bezier is all there is right now. From there you use Bezier as before.

fr0sty wrote on 2/24/2023, 10:14 PM

Mindmatter wrote on 2/2/2023, 9:09 PM

If I understand this feature correctly, it should be able to replace green screen keying at some point, right? I suppose it should be pretty easy when the subject is clearly distinct from a flat background.

Unfortunately based on the Resolve experience, the answer is No. it is not a replacement for a green screen. The reason is you can perfect a single frame, but when the tracking occurs you can get a different result on every frame, assuming a moving subject. It just depends if you would not be embarrassed by a Zoom meeting type background replacement in the video you were producing. Repeatedly with Resolve I've had to ditch their AI masking because if it's not perfect it looks amateurish.

Color grading is a better task, say you wanted to alter exposure on your subject but not affect your background, or to separate your subject from a background maybe you reduce exposure or blur the background. Badly tracked frames aren't' as obvious when it's not a background/foreground replacement but a color grade instead. @Mindmatter

This is especially true if you slightly feather the edges of your mask, further hiding any imperfections.

IveTechMedia wrote on 5/3/2023, 5:13 PM

Can anyone explain how I get the masks created to work with motion. Probably doing something wrong but each time I create a new mask on the timeline with Smart Mask and send it to the Bezier Mask it goes to a new instance of the Bezier mask. I'm assuming I want all the masks I create to go to the same instance of the Bezier Mask so they can be added as Keyframes for motion tracking? The mask I'm tracking changes in shape so I need keyframes in the single Bezier Mask to track motion and shape. Or is this just a missing function at the moment with only single masks possible and no motion tracking of changing masks created by Smart Mask? Thanks, hope this question is clear enough!
Oh should have mentioned I tried adding keyframes in Smart Mask before sending to Bezier Mask but the keyframes just disappear when I analyse a new section of the video. Hope this clarifies further

Former user wrote on 5/3/2023, 6:22 PM

@IveTechMedia I think there's a bit of false advertising, this post https://www.vegascreativesoftware.com/gb/new-features/#productMenu on New features says 'Create and apply multiple masks to a video frame with multiple objects, each with its own masking controls', - After Analysing, hold the Ctrl key down & select different objects/people, then Create Mask will put them in the same Bezier tab,

but the video in the link suggests it'll flow with the motion of the people dancing, the only way that would happen is with hundreds of masks or the mask keyframed ?

If I'm wrong I'd like to know how it is done?

JamBam wrote on 5/9/2023, 12:45 AM

@Former user I agree with you. They've definitely given the impression of a full rotoscoping solution, which looks possible with the tech they've demonstrated here, but the smart mask only captures one frame. Even though the mask is quite accurate, It's totally impractical to manually modify it mask frame-by-frame to make it flow with the identified objects since so many anchor points are created with the AI engine. I haven't figured out a way to create separate masks and link them frame by frame to give the animated result like they've illustrated in their video. It's possible to use the track motion feature of the beizier mask but that won't change the shape of the mask with the object so it's not that useful unless your subject never changes shape. And unfortunately, unless they are residents of South Park, people are shape-shifters. I've also found it a bit crashy, but still, it's a work-in-progress and it's possible to see where they are headed from here. I would definitely encourage more development - there's a long way to go to match where Davinci Resolve is with their magic mask. Would love that capability though.