Upscaling Quality

wwaag wrote on 6/19/2020, 3:46 PM

From time to time, most editors have a need to upscale or uprez some of their video clips. It can be done in Vegas by simply rendering using a higher resolution template with/or without use of the Smart Upscale Fx. There are also many external open-source resizers that are available. The question becomes--how good is the upscaling? Most editors simply use the “eyeball” approach which is perfectly OK.

However, like render quality (RQ), some may want to see more evidence-based objective measures since “eyeballs” are known to differ dramatically. Here’s an approach that makes use of the RenderQuality tool that is freely available in the HappyOtterScripts Free Tools library. https://tools4vegas.com/library/

The idea is very simple. Here are the steps:

1. Begin with some “full” resolution video footage.
2. “Down rez” that footage to the same aspect ratio and frame rate using a high quality resizer--or any resizer for matter.
3. Then "upscale” the "down-rezzed" video to the original frame size, again keeping the same aspect ratio and frame rate.
4. Compare the quality of the original “full” resolution video footage with the “upscaled” video using the RQ tool.
5. For all renders, it is important to use a lossless codec such as MagicYUV, UtVideo or Uncompressed.

For the following tests, the footage I used may be downloaded here. https://www.dropbox.com/sh/8fzp947zo2eijnf/AADcbXXw5AxNYrCN96I2brvZa?dl=0
The original “out-of-the-camera” file is Butterfly.MTS which is FHD at 60P. The down-rezzed test file is UpRezTestClip.avi which is 854 x 480 at 60P. It was created using the Lanczos-4 resizer with MagicYUV RGB as the format. As you can see, there is a fair amount of motion content in the test video.

Here’s a demo of the procedure used for one of the conditions--a straight render from Vegas using an FHD render template.

Resizers included those available in HappyOtterScripts plus the “super-resolution” resizer that is available at http://www.infognition.com/. I should note that Video Enhancer which includes the “super-resolution” resizer is a standalone app that is now freeware. Moreover, the resizers used in HOS are also open-source and are freely available.

And here are the results which more or less speak for themselves.

From the results, the two Lanczos resizers resulted in the smallest error, although the Spline, Blackman, and SuperRes resizers were not far behind in terms of quality. Again, all of these are open-source and can be freely downloaded.

It should be pointed out that the Vegas Smart Upscale Fx should not be judged in terms of its error alone since I suspect that the Fx does some noise filtering and sharpening which would always increase the measured error regardless of the resizer being used. To judge its quality, one has to resort back to the “eyeball” method, which as I stated at the outset, is perfectly OK. The only truly valid results are the Vegas-480 and Vegas-1080 (different project frame sizes) which were identical as expected.

Comments are welcome.

Comments

JN- wrote on 6/20/2020, 7:06 AM

@wwaag So, is there a demo available to show using the top of the table LanczosHOS resizer?

JN- wrote on 6/20/2020, 8:00 AM

@wwaag I tried using HOS on my laptop with the latest update, still has problems, cannot use it.

I selected to do a clean install of HOS, remove registry entries etc. I then had to add my serial no. again.

Despite this still doesn't work. I was having a go at the upscaling you mentioned in this thread.

I followed the on screen error message and did a script reset but still the same.

This is what I was attempting, pretty basic I would think.

JN- wrote on 6/20/2020, 8:49 AM

@wwaag So I attempted using HOS on my PC, where HOS was previously working ok, just the laptop had problems. I have recently updated both machines to WIN 2004.

I updated HOS on the PC without a clean HOS install, just update.

I immediately got an error message when I clicked on the R+ item ...

I OK'd that and continued to attempt to use HOS, but then VP17 just bombed out completely, haven't had that happen in quite some time. I've been using VP17 recently with several small projects, 640 x 480 x 30 fps video + stills, using vp not HOS. When I saw the HOS update I decided to update and also try the uprez option, seeing as how its superior to whats built into vp.

I was attempting initially, using HOS, on my laptop to uprez to 1280 x 960, I had already done this using vegas aok.

wwaag wrote on 6/20/2020, 6:42 PM

@JN-

When it crashes, does it generate a crash report? If so, send it along with some identifying information so that I can contact you. If it doesn't generate a crash report, then upload the Render+ folder (or whatever tool you are using) or attach it to en email and send it to support@tools4vegas.com. Without that information, it's pretty much impossible to identify the problem. I've only received one crash report in the last 4 or 5 days from someone using 24bit audio attempting to render to flac which probably wasn't you. Let me know.

JN- wrote on 6/20/2020, 7:01 PM

“which probably wasn't you” Correct. I’ve never sent in any reports, no idea where I would send them either.

@wwaag When VP crashed, on my PC, VP offered to send on a report, I declined as I figured it was a HOS caused problem, not a VP caused problem. HOS didn’t offer to send on a report.

If HOS generates a crash report I have no idea where I would find that. I was attempting to use the R+ Avisynth settings at the time.

The only feedback from HOS was the “object reference ..... “ error dialog that I have already posted.

All of the above relates to the PC issue.

——————————————————

The laptop issue is a separate issue explained in my other post above the PC error.

Maybe if I only posted one problem at a time it would be easier.

wwaag wrote on 6/20/2020, 7:09 PM

Again, you need to send me the folder. The object reference error can result from many different things so it is not sufficient in itself to diagnose the issue. Obviously, HOS did not produce a crash report. When it does, you will see it with the option to add attachments and whatever additional information you want to provide. You just click Send and it automatically uploads the info to my DropBox account.

JN- wrote on 6/20/2020, 7:24 PM

@wwaag I remember now that I previously had that "Object reference" issue and it was fixed by copying the file Frameserver.x64.fio2007-config from the VP16 to the VP17 folder. I just checked and its already there, so its not that. The date is yesterday, so it went in with the new install probably.

Yes there was no crash report from HOS.

What folder to send? and where to send to?

I did a global search on drive C for render+ and nothing came up, you need to give me the path/location of the render+ folder.

JN- wrote on 6/20/2020, 7:47 PM

@wwaag What I did find are three folders called RenderPlus ...

C:\ProgramData\HappyOtterScripts\Magix Vegas Pro\RenderPlus\

C:\Users\All Users\HappyOtterScripts\Magix Vegas Pro\RenderPlus\

C:\Windows.old\Users\All Users\HappyOtterScripts\Magix Vegas Pro\RenderPlus\

wwaag wrote on 6/20/2020, 8:04 PM

Sorry. Folder is "C:\ProgramData\HappyOtterScripts\Magix Vegas Pro\RenderPlus". Send to support@tools4vegas.com.

JN- wrote on 6/20/2020, 8:08 PM

@wwaag Ok. I’ll send tomorrow, my PC is in outside location, 02:08 am here in the morning, thanks for getting back.

ceejay7777 wrote on 6/21/2020, 6:39 AM

... I've been using Super Scale in DaVinci Resolve recently ... how does that compare to your results?

JN- wrote on 6/21/2020, 7:49 AM

@wwaag

I sent on emails to support for both the PC and laptop errors.

RE: the laptop issue.

Using RenderPlus Avisynth ...

I've discovered that if I select one of the non-custom frame size preset's there's no problem.

Although there is an option to select "custom" frame size, when I do select custom, it causes the error as previously reported.

The media I want to resize is 640 x 480 and I tried x2 = 1280 x 960, doesn't work. I then tried x3 = 1920 x 1440, doesn't work either. What does work is the built in x2.25 = 1440 x 1080, non custom, is a drop down.. All have an AR of 1.33.

So currently my only option is using the x2.25 1440 x 1080.

Can the 1280 x 960 be added into the dropdown list to overcome this? My preference is to upscale but not excessively, so x2 seems like a good idea?

JN- wrote on 6/21/2020, 8:50 AM

@wwaag RE: The PC issue.

Once I use one of the drop downs is ok, same error message as on laptop if I enter say 1280 x 960.

So I can now use the PC also once I don't select custom frame rate in RenderPlus Avisynth ...

Maybe the crash was a once off?

Former user wrote on 6/21/2020, 9:32 AM

... I've been using Super Scale in DaVinci Resolve recently ... how does that compare to your results?

Technically i'm not sure, but using eyes you can tell the difference. If the intention is to create more detail but not cause a reduction in picture quality I think it does a good job. In this example I 2x upscaled a cow, and then zoomed in 200% to make comparison. Left is upscaled, Right is original.

Btw, why do you use Resolve instead of Vegas for upscale? For me it's the perceived efficiency in that Resolve uses 100% of my GPU and vegas doesn't.

wwaag wrote on 6/21/2020, 10:20 AM

@JN-

Haven't looked at the details from your logs, but it sounds as if the problem may have been "user error". It's a bit confusing to enter custom frame sizes but it does work. Having said that, HOS should have done a better job at exception handling. Regardless, here's a demo.

@ceejay7777

No. I have not tried Resolve nor have I tried Premiere since I don't have those apps (I should probably get both.)

JN- wrote on 6/21/2020, 11:28 AM

@wwaag OK. I thought it was because I was entering 1280 x 960 with spaces, but that isn't it, the demo helped thanks.

FWIW I earlier on eyeballed an output test clip x2.25 (not custom, built in, 1440 x 1080, Lancros4) against the VP17 MC CPU and the edges looked a fraction poorer using HOS. I'll give it a go later with the x2 1280 x 960, eyeball test and report back, great to have it working now.

JN- wrote on 6/21/2020, 2:18 PM

@wwaag

Did a comparison on what I had rendered out and upscaled (640 x 480 to 1280 x 960) in VP17 MC CPU and HOS, x264 CPU Lanczos4.

I could see absolutely no difference. Unfortunately when I checked the data rates I saw that the x264 was about half of the VP17, so of limited use I guess.

So I rendered out in HOS a 25% higher data rate and compared it to the 1st HOS piece, I could see no difference.

Former user wrote on 6/21/2020, 8:19 PM

... I've been using Super Scale in DaVinci Resolve recently ... how does that compare to your results?

I found a comparison . Davinci Resolve does poorly compared to Lanczos. The values are in VMAF which is a perceptual video quality assessment algorithm developed by Netflix

Chart1. VMAF scores of reconstructing 2x downsampled “Dinner Table” 1080p HD Shutterstock video

Chart2. VMAF scores of reconstructing 3x downsampled “Dinner Table” 1080p HD Shutterstock video

https://medium.com/@jonfryd/comparing-video-upscaling-methods-on-stock-footage-78fc81914567

fifonik wrote on 6/22/2020, 8:18 PM

Sorry for going a bit off-topic, however I'm strongly against using "single number" to compare video quality. This is really misleading.

I created two 2-frames videos (5x5 pixels each).

First:

Date: 2020/06/23  10:50:49  
Frames Processed: 2
Processing Speed: 2 fps
Mean Squared Error: 2949.12
Peak Signal to Noise Ratio: 54.712

Second:

Date: 2020/06/23  10:50:33  
Frames Processed: 2
Processing Speed: 2 fps
Mean Squared Error: 3495.453
Peak Signal to Noise Ratio: 12.708

First one have much better values, right? In math -- sure. For human -- it depend...

Here are the videos (I upscaled them to 200x200 before uploading):

When comparing quality metrics as graphs, you will notice that in the first video quality the first frame is as in original, but the second frame's quality is much worse than any frames in second video.

This is like comparing averages.

2 10 -- average 6

6 6 -- average 6 (also)

But what would be better -- it is really tricky question.

UPDATE:

Here are the graphs I build using the provided mp4 files (not original 2-frames 5x5px files as they were too small and too short and calculation fallen for some metrics) using VQMT Free:

PSNR:

SSIM:

VQMT:

VMAF:

Every metric provides similar results. My point if not about "what metric to be used", but about "averaging" frames' metrics into single value per file.

If I remember correctly, I've already suggested add to RQM tool ability to draw simple graph (this can be done now using per frame log in Excel). If for two files graphs looks similar, it would be quite OK to compare single value metrics.

Former user wrote on 6/22/2020, 10:05 PM

That's the advantage VMAF is said to have over PSNR for automated tasks without human observation. The VMAF value is a combination of

a) Anti-noise SNR (AN-SNR),
b) Detail Loss Measure (DLM),
c) Visual Information Fidelity (VIF)
d) Mean Co-Located Pixel Difference (MCPD)*

*The MCPD of a frame with respect to the previous frame, the
temporal component, is important and is often lacking in metrics
that merely compare a decoded image with a reference image.

wwaag wrote on 6/23/2020, 12:41 AM

Just read the PixOP comparison of different resizers. In looking at their charts, it is of interest that Lanczos was always at or near the top for the remaining contenders, pretty much the same as I found. I might add that VMAF is also open-source, and has now been implemented in FFmpeg much the same as PSNR and SSIM. The reason I didn't include SSIM in the comparisons is that it takes so long to compute for my implementation.

BTW. For anyone wanting to try PixOp upscaling, I found this.

JN- wrote on 6/23/2020, 3:24 AM

@wwaag Can VMAF be added into your RQ metrics tool with existing SSIM, PSNR & MSE?

wwaag wrote on 6/23/2020, 2:42 PM

@JN-

Adding VMAF would be a major undertaking and not worth the effort in my view since my suspicion (I may very well be mistaken.) is that the ordinal results would not be a lot different from other simpler measures. As your testing on RQ has shown, the results are pretty much the same even though the exact numbers may differ. For the most part, low MSE usually results in high SSIM scores and conversely low SSIM usually means higher MSE, etc. Probably the best tool at the moment for investigating RQ is VQMT--the downside is that a license that supports even FHD testing is $999.

JN- wrote on 6/23/2020, 4:13 PM

@wwaag “the results are pretty much the same even though the exact numbers may differ”

Indeed, there is little to be gained for a lot of effort, there is overall a fair bit of consistency in the results already.