Jaggy Artifacts in Rendering from Analog Source

Pea-Hicks wrote on 8/1/2022, 3:11 PM

Hi all, I'm trying to figure out Vegas Pro 18 render settings to avoid getting the jaggy artifacts you can see along the edges in this example frame:

The source is analog VHS, going into a Blackmagic Decklink Studio via S-Video. The problem is not with the original video- I get this effect no matter what analog source I try. Here's my project settings:

I've tried various progressive render settings/presets, including MainConcept AVC/AAC with the deblocking filter turned on, and nothing seems to work. Rendering interlaced looks worse, with typical interlace combing.

If anyone could share with me the exact render settings I need in order to get rid of the jagged edges, I'd really appreciate it!

Comments

Former user wrote on 8/1/2022, 3:14 PM

@Pea-Hicks are they on the orig footage when you watch it on a media player?

Pea-Hicks wrote on 8/1/2022, 3:17 PM

Hi, no, it's not the source. I get this effect on every tape I try (VHS, Betacam SP, Umatic, etc).

Musicvid wrote on 8/1/2022, 3:20 PM

In order to do this properly, you need to match your project to your source. That means Resolution, Interlacing, PAR, and Frame Rate. Then render Magix AVC Progressive the same resolution as your source.

If on the other hand, you will want to Upscale your video, you will do differently, which is to follow the Help Instructions precisely for Smart Deinterlace and Upscale.

The differences between the two procedures need better documentation and clarification; this has been reported to the development staff. Either way, you may notice slight edge combing artifacts.

Two different purposes, two entirely different procedures. Best of luck.

Pea-Hicks wrote on 8/1/2022, 3:23 PM

Hi, thanks- I've tried setting the project and render settings to match the source, and the result is worse- interlaced video with combed interlace lines. I am, actually, trying to upscale the video and convert it to progressive.

Former user wrote on 8/1/2022, 3:34 PM

@Pea-Hicks Hi, you could share a short file on Google Drive? i'm sure others will try it out n see .

Pea-Hicks wrote on 8/1/2022, 3:37 PM

I tried setting the project settings to match the source, and the Magix AVC Progressive render settings also to match the source, and the jagged edges remain,

Pea-Hicks wrote on 8/1/2022, 3:38 PM

@Pea-Hicks Hi, you could share a short file on Google Drive? i'm sure others will try it out n see .

Hi, do you mean the original capture avi?

Pea-Hicks wrote on 8/1/2022, 3:40 PM

For what it's worth, I'm using the legacy Vegas Capture Video. I've just found it to be the most intuitive video capture app for me. I don't think this is the culprit, but let me know if it might be.

Pea-Hicks wrote on 8/1/2022, 4:01 PM

OK here's a short avi of a raw capture:

https://www.dropbox.com/s/rl8qzeo0zjvn0k5/countdown%20001.avi?dl=1

here's a screenshot of the avi playing:

and here's a screenshot of a typical render out of Vegas (using all the settings I outlined in my original post- the jaggies appear no matter what combination of settings I try to use).

(fyi the pixel aspect ratio of raw capture and screenshots is .9091, which ultimately I want to convert to square for the final render.)

Musicvid wrote on 8/1/2022, 4:13 PM

For what it's worth, I'm using the legacy Vegas Capture Video. I've just found it to be the most intuitive video capture app for me. I don't think this is the culprit, but let me know if it might be.

Yes, it's pretty basic technology. Handbrake Decomb has better results, but like I said, expect imperfect results from any method. You can take that for fact.

You can see my results for the Smart Deinterlace and Upscale method in my Signature by clicking below. It's my DVCAM Highlight Reel from 2000-2010 upscaled to 720p.

 

fr0sty wrote on 8/1/2022, 4:41 PM

Those artifacts are from the source video being interlaced, which is common of all analog video from the SD era. You need to de-interlace it to get rid of that. Set the field order to progressive in project settings and turn on smart adaptive GPU de-interlacing.

Systems:

Desktop

AMD Ryzen 7 1800x 8 core 16 thread at stock speed

64GB 3000mhz DDR4

Geforce RTX 3090

Windows 10

Laptop:

ASUS Zenbook Pro Duo 32GB (9980HK CPU, RTX 2060 GPU, dual 4K touch screens, main one OLED HDR)

Pea-Hicks wrote on 8/1/2022, 4:48 PM

Those artifacts are from the source video being interlaced, which is common of all analog video from the SD era. You need to de-interlace it to get rid of that. Set the field order to progressive in project settings and turn on smart adaptive GPU de-interlacing.

Hi, I’ve done all of those things (see my original post). The problem remains no matter what project/render settings I use.

Pea-Hicks wrote on 8/1/2022, 4:51 PM

For what it's worth, I'm using the legacy Vegas Capture Video. I've just found it to be the most intuitive video capture app for me. I don't think this is the culprit, but let me know if it might be.

Yes, it's pretty basic technology. Handbrake Decomb has better results, but like I said, expect imperfect results from any method. You can take that for fact.

You can see my results for the Smart Deinterlace and Upscale method in my Signature by clicking below. It's my DVCAM Highlight Reel from 2000-2010 upscaled to 720p.

 

Can you point me directly to the “Smart Deinterlace and Upscale method” you’re referring to?

fr0sty wrote on 8/1/2022, 5:15 PM

Smart deinterlacing can be enabled in project settings. Upscale is an effect you add to the video. Once you add it, you must drag it before "event pan/crop" in your effects chain for it to activate (at the top of the effects window when you add it). Using the smooth upscale mode produces the fastest results, the sharp modes take forever to process and don't always yield better results. The new upscaler uses artificial intelligence to help it better draw the pixels, it knows what a human is, a tree, etc... and can use that info to aid it when redrawing the pixels.

Last changed by fr0sty on 8/1/2022, 5:16 PM, changed a total of 1 times.

Systems:

Desktop

AMD Ryzen 7 1800x 8 core 16 thread at stock speed

64GB 3000mhz DDR4

Geforce RTX 3090

Windows 10

Laptop:

ASUS Zenbook Pro Duo 32GB (9980HK CPU, RTX 2060 GPU, dual 4K touch screens, main one OLED HDR)

Pea-Hicks wrote on 8/1/2022, 5:23 PM

Smart deinterlacing can be enabled in project settings. Upscale is an effect you add to the video. Once you add it, you must drag it before "event pan/crop" in your effects chain for it to activate (at the top of the effects window when you add it). Using the smooth upscale mode produces the fastest results, the sharp modes take forever to process and don't always yield better results. The new upscaler uses artificial intelligence to help it better draw the pixels, it knows what a human is, a tree, etc... and can use that info to aid it when redrawing the pixels.

OK, so yeah- as per my original post, I always have "Smart Adaptive (GPU Only)" selected in the project settings. As for the "Smart Upscale" effect, I know I've tried it before with no perceivable effect, but I'll experiment with it again.

Pea-Hicks wrote on 8/1/2022, 5:28 PM

OK, I tried the smart upscale plugin, and it made no difference. yes, i did put it before pan/crop.

Pea-Hicks wrote on 8/1/2022, 5:31 PM

I just realized that Vegas 19 has a new upscale plugin. I'm using Vegas 18. Maybe time for me to upgrade, i guess, but hopefully there's a way I can do this without spending more money...

fr0sty wrote on 8/1/2022, 5:44 PM

No, upgrades to new versions cost. Not very much (I think it's under $200), but they do cost. VEGAS 20 is on the verge of releasing, and there's a deal going now where if you buy 19, you get 20 also.

Systems:

Desktop

AMD Ryzen 7 1800x 8 core 16 thread at stock speed

64GB 3000mhz DDR4

Geforce RTX 3090

Windows 10

Laptop:

ASUS Zenbook Pro Duo 32GB (9980HK CPU, RTX 2060 GPU, dual 4K touch screens, main one OLED HDR)

Pea-Hicks wrote on 8/1/2022, 6:01 PM

OK stop press, I think I may have found a workaround, with a couple caveats. Topaz Video Enhance AI seems to do the trick, BUT... it takes FOREVER to render, and it renders without audio. It also only seems to render at a .9091 pixel aspect... so I have to take the Topaz render (1080p), bring it into Vegas, set the pixel aspect to square in the media properties, drop in the audio from the raw capture, and then render normally. I'll post an example here shortly. I'll try out Vegas 19 and see if the improved upscale plugin give similar results to Topaz, and if it does, without a huge render time, that'll be reason enough for me to spend the $$$ I guess.

fr0sty wrote on 8/1/2022, 6:05 PM

Don't expect superior or even equal results without the render time, but expect similar, without the need to leave VEGAS.

Pea-Hicks wrote on 8/1/2022, 6:38 PM

OK, here's an example using Topaz as outline above. This is basically the type of result I was looking for:

https://www.dropbox.com/s/pbca779s5loqvu1/MTRP%20Overview%20-%20Barbara%20Nadkins.mp4?dl=1

I've just bought the Vegas 19/20 upgrade, so I'll play around with that and hopefully I can get a similar result with less hassle.

Pea-Hicks wrote on 8/1/2022, 7:01 PM

welp... just upgraded to Vegas 19 and... the program won't even start. Neither will Sound Forge 15. I give up!

Musicvid wrote on 8/1/2022, 7:10 PM

I did comprehensive testing of VP19 Smart Deinterlace and Upscale vs. Topaz back in May. I was not allowed to publish the results here because I used a short sample of Star Wars footage in my investigation, which I believed passed muster for educational fair use.

However, I can share the outcome.

  • Vegas 19 has slightly better Spatial (still frame) detail.
  • Topaz has slightly better Temporal (motion) detail.
  • Topaz took 4x longer to do its job than Vegas did (with light Unsharp Mask preceding Upscale, but without the AI Sharpen Model).
  • For that reason, I concluded that Vegas is best for my work; it is relatively "less slow" than Topaz, by a country mile.
  • Your uploaded video looks pretty good. There is a tendency to overprocess and oversharpen when using Topaz, because the lenses and sensors and tape heads in those days were shitty.
Pea-Hicks wrote on 8/1/2022, 8:32 PM

OK, so I managed to get Vegas 19 to start (it was hanging on some janky Magix ASIO driver that got installed with Sound Forge Audio Studio 15, an app I don't actually need, so I uninstalled that and now Vegas boots fine). I tested all three modes of the new upscale plugin, and none of them eliminated the jagged edges. Unless I'm doing something wrong (yes, i put the plugin before pan/crop). Oh well, hopefully there's some other benefits for me to offset that upgrade fee... guess I should have just tried the demo instead! Lesson learned... back to Topaz, I guess, though with these render times, it's probably gonna take a week of straight rendering to do all the videos I'm hoping to do...