Fixing blemishes in sky using AI or any other means

Len Kaufman wrote on 7/28/2024, 1:18 PM

Hi all,

I have a problem video clip that is important to a project I am working on. The sky in the clip has a "blemish" (see attached image) which is quite noxious. It's bad enough when the camera is still, It is really problematic when the camera moves, as it calls more attention to the blemish. I believe the blemish was caused by condensation on the lens. After all, it was shot in India, and temperature was over 100 F (real degrees, not the "feels like" temp). It is even more apparent, when I adjust the colors, sharpness, and saturation in the clip. But if I don't adjust those 3 factors, the clip is not work using. If I need to let the sky go to solid white, that wouldn't be terrible, but not best outcome.

I tried to use AI, as found in Vegas Pro 21, latest version, but the only thing the AI seems to find is the guy who is demonstrating how to wind a turban.

Is there a way to get the AI to "find" the sky, and work with that? Though I've been using Vegas Pro for many years, I'm quite new to AI. If AI is not the answer, any other means? Reducing the "confidence" slider finds lots of stuff, but not the sky.

Thank you.

Comments

DMT3 wrote on 7/28/2024, 1:30 PM

You might be able to replace the sky by using chromakey. There doesn't seem to be much other blue and the blemish looks close enough you should be able to trim the chromakey to include. Then get a generic shot of some clouds and replace it.

Len Kaufman wrote on 7/28/2024, 1:46 PM

@DMT3 Thank you for your response. I should have mentioned that I had tried Chromakey earlier, but there is enough "black" or "gray" in there to continue to show. I was hoping to find some "magic" (Magix?LOL) in AI.

Trying it again now. Rendering as I write. I'll let you know.

Len Kaufman wrote on 7/28/2024, 2:01 PM

@DMT3 No, that didn't work. I even tried it with the eye drop sampler "pointed at the dark area of the blue.

DMT3 wrote on 7/28/2024, 2:14 PM

I guess the camera is moving?

Len Kaufman wrote on 7/28/2024, 2:27 PM

@DMT3 Yes, very much so. Hand held, physically moved in, and I zoom into closeup shot of the guy winding the turban. After I zoom in, the sky is not showing much, so I'm OK there. The angle of the sun in relation to the camera didn't help much either. There is one point where the condensation catches the sun and creates another light orb artifact into the lower picture area, against the building, but it is brief and I can edit that small portion out without too much trouble.

Yes, I know to check for condensation on the lens going from cool to warm (I live in Florida), but this guy's demo wasn't specifically for me. He was starting and I had to grab it while it was happening.

I can fix all of these in Photoshop or Lightroom....let's see....1min20sec x 60 fps=4800 frames.....ugh. That's why I was hoping the magic of AI may play a part in the solution. Are there any specifically AI programs that might do such a thing?

DMT3 wrote on 7/28/2024, 2:29 PM

@Len Kaufman
I did a test on the still using mask and chromakey. I got a pretty good key and depending on the amount of movement, you might be able to track the mask

Len Kaufman wrote on 7/28/2024, 2:50 PM

@DMT3 Thank you. I'll try that later today. When you used Chromakey, where did you take your sample? From the dark blemished area or from the general blue sky?

Gid wrote on 7/28/2024, 2:53 PM

@Len Kaufman Hi, do you have Vegas Mocha? You might be able to overlay a cleanplate image, sort of similar to what you would have to do in Photoshop except it'll track & auto overlay each frame.

I've got Mocha Pro not Vegas Mocha so I'm not sure how you would go about this with Vegas Mocha or even if it's poss practically speaking. 🤷‍♂️

Can you show the full '1min20sec' video so the camera movement can be seen?

Len Kaufman wrote on 7/28/2024, 3:00 PM

@Gid @DMT3 Yes, I have Vegas Mocha. Never used it before, so whole 'nother learning curve. But I will give it a go. Thank you for the suggestion.

I guess I can post it to YT, as only visible to someone with link and post the link here. Was hesitant to post it making download possible, though I guess there are ways of downloading even those videos so designated. Maybe I'll post it with a watermark.

DMT3 wrote on 7/28/2024, 3:04 PM

I sampled it, but then kept moving the color with mask only on until I was able to mostly only mask the sky.

Len Kaufman wrote on 7/28/2024, 3:07 PM

@DMT3 Thank you for taking the time. In all the years of using Vegas Pro (since it was a single digit release.. 08?), I've never felt compelled to do masking. I guess that's about to change. But where does AI fit into this? Not at all? You would think if AI can create a human hand with 6 fingers, it should be able to fix a sky! LOL😎

DMT3 wrote on 7/28/2024, 3:08 PM

I don't have access or know much about AI. I will try to screen shot what I did. Give me a few minutes.

Len Kaufman wrote on 7/28/2024, 3:15 PM

@DMT3 @DMT3 First thing that popped up when I launched Vegas Mocha is a question in "Daily Tips," asking if I've had a desk break today, and suggesting I do so. So, I will do that before tackling Mocha...though I may go to the kitchen and have some "mocha" there.

DMT3 wrote on 7/28/2024, 3:25 PM

@Len Kaufman Here is my process, although a bit clumsy. Enjoy your break.

Len Kaufman wrote on 7/28/2024, 3:30 PM

@DMT3 Thank you. I'll give it a go.

mark-y wrote on 7/28/2024, 10:13 PM

In Vegas 21, you might try Z-Depth. TBH, it looks like smoke, not your lens.

Len Kaufman wrote on 7/28/2024, 10:22 PM

@mark-y Thank you. I was unaware of that function, and I just looked it up. I will give it a shot, as well as the above suggestions. Whatever the cause of the dark area, smoke or something else, if I can get Z-Depth to "think" it's in the distance, I would be happy to do so.

MikeS wrote on 7/29/2024, 6:09 AM

Sky replacement with Mocha Vegas

Reyfox wrote on 7/29/2024, 6:56 AM

I had a smudge on my lens that I just didn't notice while shooting a video that had blue sky above the ruins. I used "cookie cutter" on the smudge, placed a still image of the same scene under it and found a clean part of the sky, and shifted it with Pan/Crop. I feathered the mask and a slight blur. Not noticeable at all in the final video.

Dexcon wrote on 7/29/2024, 7:46 AM

Try Vegas Pro's native Color Corrector (Secondary) FX using the following settings:

Whether or not it works on a moving video is another issue. While it may not completely eliminate the blemish, in the still image it certainly vastly reduces it.

Len Kaufman wrote on 7/29/2024, 8:18 AM

@Dexcon Thank you for the suggestion. Tried that technique initially, but it didn't solve it completely for me. While it did help in some areas, there are other parts of the clip where it didn't do enough.

Len Kaufman wrote on 7/29/2024, 8:22 AM

@MikeS Thank you for that link. I watched the video, and I can see the potential for that technique, which I will try today. I have a collections of skies that I collect on my daily walks along the beach. I collected them for use with still images, but I can see them being called into play here. I wish the folks at Boris put more effort into the audio on their videos. I'm listening through studio monitors, and the audio is less than clear.

Dexcon wrote on 7/29/2024, 8:22 AM

While it did help in some areas, there are other parts of the clip where it didn't do enough.

Would keyframing the CC (Secondary) levels help?

Reyfox wrote on 7/29/2024, 8:24 AM

@Len Kaufman you might have to combine several techniques. It doesn't look like a "one button" fix. I used Cookie Cutter, but mine was only a "spot" so it was relatively easy to fix.