Smart Mask (AI assisted automatic masking) Introduction

fr0sty wrote on 1/23/2023, 12:03 PM

This is an incomplete feature that is being released in Beta form. We need your feedback to continue its development.

Here's his introduction written by VEGASPascal for the new smart mask feature. Please check it out, let us know what works and what doesn't for you! New updates will continue to be added to improve this functionality as it comes out of beta in the future.

Hi Community,

This is a very early drop of the "Smart Mask" feature. The status of the plugin is "work in progress" but we want you to give a first impression of the feature.

Here some quick notes:

this is only an overlay plugin... nothing happens in the render process (yet)

the goal is to create a mask the user can send to the Bezièr plugin

AiA plugin (artificial intelligence assistant): user is still able to improve bad Ai results in the edit mode of Bezièr

it is a CPU only plugin (GPU precision was too bad)

new overlay function of VEGAS (OFX draw suite)

known bug: After the first time you press the "Analyse" button nothing happens in the preview. You have to restart VEGAS and start again.

Info for the plugin

Analyse: Starts the analyse process (object detection) for the current frame

ShowLabels: Shows the confidence score and the class label for each object. (only visible after analyse process)

Confidence: Filter for objects with a given confidence score (only visible after analyse process)

Shape: Improve the contour of the selected shape (only visible after selecting an object)

Transfer: Send selected shape to Bezièr plugin (only visible after selecting an object)

Boolean parameter in timeline shows the analysed frame (only visible after analyse process)

We hope you enjoy the plugin.

 

Last changed by fr0sty

Systems:

Desktop

AMD Ryzen 7 1800x 8 core 16 thread at stock speed

64GB 3000mhz DDR4

Geforce RTX 3090

Windows 10

Laptop:

ASUS Zenbook Pro Duo 32GB (9980HK CPU, RTX 2060 GPU, dual 4K touch screens, main one OLED HDR)

Comments

Cmin7669 wrote on 1/23/2023, 5:58 PM

I wonder if the Vegas team is headed down the right track here, I have recently looked at the Da Vinchi Resolve's new masking tool and ask is this the better way to go. Just wondering, maybe Vegas will get this right. I have worked with the "Trace Mask" with the Pen Tool in After Effects and that is very good also.

.

I will keep my fingers crossed for this plugin, as the possibilities are endless.

fr0sty wrote on 1/23/2023, 7:20 PM

It's literally the same thing, only that VEGAS' plugin isn't complete yet, it's still in beta form. Many planned features are not implemented yet.

Last changed by fr0sty on 1/23/2023, 7:20 PM, changed a total of 1 times.

Systems:

Desktop

AMD Ryzen 7 1800x 8 core 16 thread at stock speed

64GB 3000mhz DDR4

Geforce RTX 3090

Windows 10

Laptop:

ASUS Zenbook Pro Duo 32GB (9980HK CPU, RTX 2060 GPU, dual 4K touch screens, main one OLED HDR)

Cmin7669 wrote on 1/23/2023, 10:22 PM

Well I had a fiddle with it and maybe it's me but found it hard to get it to do anything useful at all. But as I said I'll keep my fingers crossed!

The DaV Resolve method is very intuitive and simple to use, I only hope Vegas takes the same approach and not over complicate it.

fr0sty wrote on 1/23/2023, 11:04 PM

There are numerous UI improvements planned, this is just the raw infrastructure that makes it work being tested. It will improve, and your feedback is important to help with that.

Systems:

Desktop

AMD Ryzen 7 1800x 8 core 16 thread at stock speed

64GB 3000mhz DDR4

Geforce RTX 3090

Windows 10

Laptop:

ASUS Zenbook Pro Duo 32GB (9980HK CPU, RTX 2060 GPU, dual 4K touch screens, main one OLED HDR)

paul-marshall wrote on 1/24/2023, 6:21 AM

Potentially very useful. Got it to detect and motion track a teddy bear that just happened to be in the first clip I tried, but disappointed that the AI could not recognise a vehicle registration plate. Increasingly I need to blur out a car number and that always seems very tedious.
It would identify the car and I was very impressed by how it could identify a face inside the car even though the contrast (and resolution at this point) through the windscreen was very low. There was one problem - I could not select the person inside the car so a way is need to step through 'nested' objects. Very promising start though.

Last changed by paul-marshall on 1/24/2023, 11:13 AM, changed a total of 3 times.

Windows 11.0 (64-bit)
Intel® Core™ i9 Eight-Core Processor i9-11900K (3.5GHz) 16MB Cache
Motherboard GIGABYTE Z590 UD AC (C (LGA1200, USB 3.2)
64GB Corsair VENGEANCE DDR4 3200MHz (4 x 16GB)
GPU Nvidia GEFORCE RTX3060Ti
I/O drives: Intel SSD PEKNU020TZ 2TB, Samsung SSD 870 EVO 1TB, Samsung SSD 870 EVO 1TB
SEAGATE BARRACUDA SATA-III 3.5" HDD, 2TB, 6GB/s, 7200RPM, 256MB CACHE
Audio: Soundblaster Z SE
Cameras: Sony AX-700, A7-IV, RX10-II
Vegas Po 21 latest version. Vegas user since V10

 

Roger Bansemer wrote on 2/1/2023, 11:06 AM

I'm still getting this message when I put the "smart mask" on a clip.

Sure would like to use it and try it out.

j-v wrote on 2/1/2023, 12:45 PM

@Roger Bansemer
I see the same when "VEGAS_Deep_Learning_Models_20.4.0.0_INT_221216_08-39" is not installed.

met vriendelijke groet
Marten

Camera : Pan X900, GoPro Hero7 Hero Black, DJI Osmo Pocket, Samsung Galaxy A8
Desktop :MB Gigabyte Z390M, W11 home version 23H2, i7 9700 4.7Ghz,16 DDR4 GB RAM, Gef. GTX 1660 Ti with driver
522.22 Studiodriver and Intel HD graphics 630 with driver 31.0.101.2127
Laptop  :Asus ROG Str G712L, W11 home version 23H2, CPU i7-10875H, 16 GB RAM, NVIDIA GeForce RTX 2070 with Studiodriver 522.22 and Intel UHD Graphics 630 with driver 31.0.101.2127
Vegas software: VP 10 to 21 and VMS(pl) 10,12 to 17.
TV      :LG 4K 55EG960V

My slogan is: BE OR BECOME A STEM CELL DONOR!!! (because it saved my life in 2016)

 

Roger Bansemer wrote on 2/1/2023, 12:52 PM

I found in a Pinterest website that I need to install something else for it to work but have no idea where or what. My vegas however is up to date.

j-v wrote on 2/1/2023, 1:20 PM

Did you install these both programs?

met vriendelijke groet
Marten

Camera : Pan X900, GoPro Hero7 Hero Black, DJI Osmo Pocket, Samsung Galaxy A8
Desktop :MB Gigabyte Z390M, W11 home version 23H2, i7 9700 4.7Ghz,16 DDR4 GB RAM, Gef. GTX 1660 Ti with driver
522.22 Studiodriver and Intel HD graphics 630 with driver 31.0.101.2127
Laptop  :Asus ROG Str G712L, W11 home version 23H2, CPU i7-10875H, 16 GB RAM, NVIDIA GeForce RTX 2070 with Studiodriver 522.22 and Intel UHD Graphics 630 with driver 31.0.101.2127
Vegas software: VP 10 to 21 and VMS(pl) 10,12 to 17.
TV      :LG 4K 55EG960V

My slogan is: BE OR BECOME A STEM CELL DONOR!!! (because it saved my life in 2016)

 

fr0sty wrote on 2/1/2023, 3:56 PM

Potentially very useful. Got it to detect and motion track a teddy bear that just happened to be in the first clip I tried, but disappointed that the AI could not recognise a vehicle registration plate. Increasingly I need to blur out a car number and that always seems very tedious.
It would identify the car and I was very impressed by how it could identify a face inside the car even though the contrast (and resolution at this point) through the windscreen was very low. There was one problem - I could not select the person inside the car so a way is need to step through 'nested' objects. Very promising start though.

You're better off using VEGAS Motion Tracking to do that, it's better designed for motion tracking non-huma/animal objects. Tutorial here:

Happy editing!

Last changed by fr0sty on 2/1/2023, 3:57 PM, changed a total of 2 times.

Systems:

Desktop

AMD Ryzen 7 1800x 8 core 16 thread at stock speed

64GB 3000mhz DDR4

Geforce RTX 3090

Windows 10

Laptop:

ASUS Zenbook Pro Duo 32GB (9980HK CPU, RTX 2060 GPU, dual 4K touch screens, main one OLED HDR)

Former user wrote on 2/1/2023, 5:53 PM

 

I wonder if the Vegas team is headed down the right track here, I have recently looked at the Da Vinchi Resolve's new masking tool and ask is this the better way to go. Just wondering, maybe Vegas will get this right.

I haven't looked at this yet, I recall Resolve started with only person detect, ironed out bugs, then released a new version that detected everything. So an iterative upgrade. Haven't used this yet, or looked at the frosty tutorial but it sounds like they've started out trying to detect everything even in this alpha/beta version. I"ll have a look at it later to verify this and see what it can do.

I like the idea of the number plate blurring, a requirement either in all EU countries or only the very privacy conscious ones like Germany. That feature is available in other NLE's / upscalers

Former user wrote on 2/1/2023, 6:38 PM

Had a look, not very useful, the first feature based on face recognition that probably would have worked a lot better is to detect people in clips, then sort the faces for ease of search when editing. No doubt this will be a feature in the future

fr0sty wrote on 2/2/2023, 12:18 AM

 

I wonder if the Vegas team is headed down the right track here, I have recently looked at the Da Vinchi Resolve's new masking tool and ask is this the better way to go. Just wondering, maybe Vegas will get this right.

I haven't looked at this yet, I recall Resolve started with only person detect, ironed out bugs, then released a new version that detected everything. So an iterative upgrade. Haven't used this yet, or looked at the frosty tutorial but it sounds like they've started out trying to detect everything even in this alpha/beta version. I"ll have a look at it later to verify this and see what it can do.

I like the idea of the number plate blurring, a requirement either in all EU countries or only the very privacy conscious ones like Germany. That feature is available in other

 

 

 

The smart mask effect is a mask generation effect, it isn't meant for motion tracking the same way. The Vegas motion tracking system is. What you want for license plate. Blurring is motion tracking, not auto masking. The tutorial I posted shows how to use the Vegas motion tracking system, which has been in place for years, to do exactly that.

Please keep in mind in that tutorial. I did not approach it from the perspective of somebody that already knew how to do it, so you see me poking around and making mistakes as I go, I'm basically learning as I give the tutorial, but by the end it produces a usable result . In fact, it produces superb results once you get it working, right.

 

Systems:

Desktop

AMD Ryzen 7 1800x 8 core 16 thread at stock speed

64GB 3000mhz DDR4

Geforce RTX 3090

Windows 10

Laptop:

ASUS Zenbook Pro Duo 32GB (9980HK CPU, RTX 2060 GPU, dual 4K touch screens, main one OLED HDR)

Mindmatter wrote on 2/2/2023, 5:09 AM

If I understand this feature correctly, it should be able to replace green screen keying at some point, right? I suppose it should be pretty easy when the subject is clearly distinct from a flat background.
I sure hope so...the amount of problematic green screens I've dealt with is just really frustrating.

So far, there's "room for improvement".

Last changed by Mindmatter on 2/2/2023, 5:27 AM, changed a total of 1 times.

AMD Ryzen 9 5900X, 12x 3.7 GHz
32 GB DDR4-3200 MHz (2x16GB), Dual-Channel
NVIDIA GeForce RTX 3070, 8GB GDDR6, HDMI, DP, studio drivers
ASUS PRIME B550M-K, AMD B550, AM4, mATX
7.1 (8-chanel) Surround-Sound, Digital Audio, onboard
Samsung 970 EVO Plus 250GB, NVMe M.2 PCIe x4 SSD
be quiet! System Power 9 700W CM, 80+ Bronze, modular
2x WD red 6TB
2x Samsung 2TB SSD

Former user wrote on 2/2/2023, 6:40 AM

Mindmatter wrote on 2/2/2023, 9:09 PM

If I understand this feature correctly, it should be able to replace green screen keying at some point, right? I suppose it should be pretty easy when the subject is clearly distinct from a flat background.

Unfortunately based on the Resolve experience, the answer is No. it is not a replacement for a green screen. The reason is you can perfect a single frame, but when the tracking occurs you can get a different result on every frame, assuming a moving subject. It just depends if you would not be embarrassed by a Zoom meeting type background replacement in the video you were producing. Repeatedly with Resolve I've had to ditch their AI masking because if it's not perfect it looks amateurish.

Color grading is a better task, say you wanted to alter exposure on your subject but not affect your background, or to separate your subject from a background maybe you reduce exposure or blur the background. Badly tracked frames aren't' as obvious when it's not a background/foreground replacement but a color grade instead. @Mindmatter

Roger Bansemer wrote on 2/2/2023, 7:54 AM

Did you install these both programs?

On my "Help" "Software updates" it says there are no updates to install. Where are you finding this update? Thanks

j-v wrote on 2/2/2023, 8:18 AM

Where are you finding this update? Thanks

On my Magix Account, but it is also able to download it through this link : https://www.vegascreativesoftware.info/us/forum/vegas-pro-20-build-326--139239/
It is not a part of the Vegas installation itself but by installing you are able to use some new features in the program.

 

met vriendelijke groet
Marten

Camera : Pan X900, GoPro Hero7 Hero Black, DJI Osmo Pocket, Samsung Galaxy A8
Desktop :MB Gigabyte Z390M, W11 home version 23H2, i7 9700 4.7Ghz,16 DDR4 GB RAM, Gef. GTX 1660 Ti with driver
522.22 Studiodriver and Intel HD graphics 630 with driver 31.0.101.2127
Laptop  :Asus ROG Str G712L, W11 home version 23H2, CPU i7-10875H, 16 GB RAM, NVIDIA GeForce RTX 2070 with Studiodriver 522.22 and Intel UHD Graphics 630 with driver 31.0.101.2127
Vegas software: VP 10 to 21 and VMS(pl) 10,12 to 17.
TV      :LG 4K 55EG960V

My slogan is: BE OR BECOME A STEM CELL DONOR!!! (because it saved my life in 2016)

 

Howard-Vigorita wrote on 2/2/2023, 11:28 AM

@Roger Bansemer Another way to download it is with the direct link in Nick Hope's "MORE IMPORTANT POSTS - Please read this first!" which is pinned near the top of the Forum. It's the one under that titled: "Where can I download VEGAS Pro and other VEGAS software?" ... the Download Manager links are more foolproof but the individual direct links are quicker if you are careful to get the right DLM (20.4.0.0) that goes with Vegas build 326. Also be aware that if you already have that DLM installed, running the install again will remove it and you'll need to run it yet again to get it back. You can look in Windows/Settings/Apps to see if it's already installed.

Roger Bansemer wrote on 2/3/2023, 6:44 AM

Where are you finding this update? Thanks

On my Magix Account, but it is also able to download it through this link : https://www.vegascreativesoftware.info/us/forum/vegas-pro-20-build-326--139239/
It is not a part of the Vegas installation itself but by installing you are able to use some new features in the program.

Thanks. I downloaded the update and now it appears to recognize objects with a vague mask. In addition to just recognizing objects, is there a way to apply that mask so as for instance to adjust the color etc on the mask? Not sure how to use it.

 

Mindmatter wrote on 2/3/2023, 11:57 AM

Thanks Todd!

I really hope that at some point, someone will come up with a better, AI based solution for green screens, or any type of similar masking jobs. Although I've done dozens of green screens , in any possible constellation, lighting , camera and studio quality and circumstance, I've yet to come across one that isn't really tedious to properly key out.

 
 

Unfortunately based on the Resolve experience, the answer is No. it is not a replacement for a green screen. The reason is you can perfect a single frame, but when the tracking occurs you can get a different result on every frame, assuming a moving subject. It just depends if you would not be embarrassed by a Zoom meeting type background replacement in the video you were producing. Repeatedly with Resolve I've had to ditch their AI masking because if it's not perfect it looks amateurish.

Color grading is a better task, say you wanted to alter exposure on your subject but not affect your background, or to separate your subject from a background maybe you reduce exposure or blur the background. Badly tracked frames aren't' as obvious when it's not a background/foreground replacement but a color grade instead. @Mindmatter

 

AMD Ryzen 9 5900X, 12x 3.7 GHz
32 GB DDR4-3200 MHz (2x16GB), Dual-Channel
NVIDIA GeForce RTX 3070, 8GB GDDR6, HDMI, DP, studio drivers
ASUS PRIME B550M-K, AMD B550, AM4, mATX
7.1 (8-chanel) Surround-Sound, Digital Audio, onboard
Samsung 970 EVO Plus 250GB, NVMe M.2 PCIe x4 SSD
be quiet! System Power 9 700W CM, 80+ Bronze, modular
2x WD red 6TB
2x Samsung 2TB SSD

RogerS wrote on 2/4/2023, 7:44 AM

In the bezier mask you can choose to mask fx which is useful for color correcting a selected area.

Custom PC (2022) Intel i5-13600K with UHD 770 iGPU with latest driver, MSI z690 Tomahawk motherboard, 64GB Corsair DDR5 5200 ram, NVIDIA 2080 Super (8GB) with latest studio driver, 2TB Hynix P41 SSD, Windows 11 Pro 64 bit

Dell XPS 15 laptop (2017) 32GB ram, NVIDIA 1050 (4GB) with latest studio driver, Intel i7-7700HQ with Intel 630 iGPU (latest available driver), dual internal SSD (1TB; 1TB), Windows 10 64 bit

VEGAS Pro 19.651
VEGAS Pro 20.411
VEGAS Pro 21.208

Try the
VEGAS 4K "sample project" benchmark: https://forms.gle/ypyrrbUghEiaf2aC7
VEGAS Pro 20 "Ad" benchmark: https://forms.gle/eErJTR87K2bbJc4Q7

Former user wrote on 2/5/2023, 6:24 AM

Thanks Todd!

I really hope that at some point, someone will come up with a better, AI based solution for green screens, or any type of similar masking jobs.

@Mindmatter I did this meme type thing with AI Mask... Now ofcourse there's a natural blue screen here, it probably wasn't needed. I AI masked the smoke, as well as the balloon. My colors are all wrong for the balloon, had night mode on, It works ok for a meme, but there's color strobing and it doesn't look real, no good for anything professional. The pixelated sky is on the camera source.

I actually did a comparison between AI Mask and Keying on a green screen with studio lighting for background replacement . Keying much better. but did not share because I'm more on the amateur side when it comes to using it, would not like to misrepresent. Once again strobing on some mask boundaries and also inability to fully remove a green outline on subject, most likely due to reflection from the green screen, much easier to remove with key

Howard-Vigorita wrote on 2/5/2023, 12:33 PM

I really hope that at some point, someone will come up with a better, AI based solution for green screens, or any type of similar masking jobs. Although I've done dozens of green screens , in any possible constellation, lighting , camera and studio quality and circumstance, I've yet to come across one that isn't really tedious to properly key out.

@Mindmatter Most of the time I don't have the opportunity to rig up a pure single-color background, like with a big green drop-cloth. But I find shooting a subject against foliage or a lawn can work just as well as chroma key to knock out a range of greens with Color Corrector (Secondary). I use it's transparency slider in combo with it's built-in hue-range masking. Been trying get AI Smart Mask to do that as easily regardless of background.

I can get a mask on a subject transferred to Bezier. But getting it to track automatically is not so easy for me. I usually end up doing it manually with keyframes. I'm probably not doing it right. Anyway, smart mask is giving me a head start. Here's what I've been able to get playing around with one of my old concert snippets:

Smart Mask is on the event and I click on Analyze. Smart Mask identifies 3 of the performers as persons pretty easily. Ha, ha, and Fred Tackett on skateboard?

So I clicked on one of the persons and fiddled with confidence and shape to get the best outline that will come up.

Then click on Create Mask and it adds Bezier Mask to the FX chain. That's the head start. It transferred it in as a curve mask whose points I can further edit to get the rest of the one performer. I slid the transparency slider over so I could also see the background while dragging the points around.

The part I don't really get is the motion tracking part. Clicked on Track Motion, got its screen, and fiddled with the controls at the bottom. I think I clicked on icons that were labeled Track both ways and transfer to Bezier which seemed to do things. Ended up looking like this:

And did this to the short clip:

Reasonably good tracking on the performer's head but not as well on the lower body where contrast was not as sharp. Maybe a little more AI here could make it easier and more effective. With a little added feathering it didn't look too bad with Style Transfer stacked behind it and overlaid over the original track.

Former user wrote on 2/5/2023, 4:36 PM

@Howard-Vigorita you were mentioning shooting infront of a uniform color to help key, did you do that here, or is the AI actually seeing a black background? In my experience black backgrounds don't work well if your subject has areas of low contrast on the boundaries (which makes sense) but I also see why someone would try with a black background, flaws in tracking not so obvious. I've given up on black/dark backgrounds so many times when using AI masks due to strobing issues. Try a well lit background and foreground. Good to see your work flow and mini tutorial