Upscaling Quality

Comments

wwaag wrote on 6/29/2020, 11:58 AM

RQM FFmpeg is now available for download at https://tools4vegas.com/library/

JN- wrote on 6/29/2020, 1:29 PM

@wwaag Thank you. I ran it on no. 27 Sony file in the FHD table posted on the previous page. The results are very similar.

ffmpeg batch file .... SSIM All .. 0.964488 (14.496229) .. PSNR Avg. .. 35.994919 .. VMAF 96.052263

RQM FFmpeg tool .. SSIM 0.9645 ....... MSE 16.3525 ....... PSNR 37.0079

Is the ffmpeg "DB" value (14.496229) similar to your calculated MSE, or a completely separate entity?

Last changed by JN- on 6/29/2020, 1:30 PM, changed a total of 2 times.

---------------------------------------------

Benchmarking thread

Codec Render Quality tables

---------------------------------------------

PC ... Corsair case, own build ...

CPU .. i9 9900K, iGpu UHD 630

Memory .. 32GB DDR4

Graphics card .. MSI RTX 2080 ti

Graphics driver .. latest studio

PSU .. Corsair 850i

Mboard .. Asus Z390 Code

 

Laptop ... (Acer Predator G9-793-77AC)

CPU .. i7-6700HQ Skylake-H

Memory ..32 GB DDR4, was previously 16 GB

Graphics card .. Nvidia GTX 1070

Graphics driver .. latest studio

wwaag wrote on 6/29/2020, 2:52 PM

No. Completely different. DB-decibels.

JN- wrote on 6/29/2020, 6:16 PM

OK, thanks.

---------------------------------------------

Benchmarking thread

Codec Render Quality tables

---------------------------------------------

PC ... Corsair case, own build ...

CPU .. i9 9900K, iGpu UHD 630

Memory .. 32GB DDR4

Graphics card .. MSI RTX 2080 ti

Graphics driver .. latest studio

PSU .. Corsair 850i

Mboard .. Asus Z390 Code

 

Laptop ... (Acer Predator G9-793-77AC)

CPU .. i7-6700HQ Skylake-H

Memory ..32 GB DDR4, was previously 16 GB

Graphics card .. Nvidia GTX 1070

Graphics driver .. latest studio

fifonik wrote on 6/29/2020, 8:31 PM

I understand the limitations of ffmpeg based versions, however it is much easier to use as almost any video format can be used.

I've also noticed some metrics inconsistencies (as @wwaag already mentioned a few times). For example, rendering using UT Video encoder from Vegas and VDub2 produces files that do not have visual or ffprobe differences but gives different ffmpeg PSNR/SSIM metrics. I will try to investigate and find solution.

P.S. I'm trying to create my own program that is calculating metrics using ffmpeg. With multiple files and "per frame" graphs. PM me (do not hijack in the thread as it is off-topic here) if you'd like to test it. Warning: this will be my first program in C# so I'd expect quite a few bugs/glitches. Here is screenshot from alpha version I have:

Last changed by fifonik on 6/30/2020, 2:12 AM, changed a total of 3 times.

Camcorder: Panasonic X920

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 3700X (not OC), RAM: G'Skill 16 GB DDR4@3200 (not OC), Graphics card: MSI RX580 8GB (factory OC), SSD: Samsung 970 Evo+ NVMe 500MB (OS), HDDs: Seagate & Toshiba 2TB, OS: Windows 10 Pro 1909

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17

JN- wrote on 6/30/2020, 1:25 AM

Looks very impressive! Wishing you the best of luck.

Don't forget the progress bar and bell/music at job completion, option to disable obviously. Simple one line per processed file log file output giving SSIM, PSNR, MSE and VMAF.

Ideally a drag and drop option into app interface for source and file(s) to process.

A counter display also indicating how many files have been processed, the number of files to process, and the current number of file that’s being processed out of total, ETA.

If you’re not using a menu/help/about dialog then show current version number permanently on screen somewhere.

Radio button selections for discrete items, SSIM, PSNR etc plus an “ALL” option.

Safe to say that thanks to my own efforts and now this contribution on your part that this thread of WWAAGS re: Uprezzing, is completely hijacked, if its important to him he's probably close to losing the will to live.

Mind you, if I don’t get a haircut soon i’ll be also.

 

Last changed by JN- on 6/30/2020, 2:32 AM, changed a total of 6 times.

---------------------------------------------

Benchmarking thread

Codec Render Quality tables

---------------------------------------------

PC ... Corsair case, own build ...

CPU .. i9 9900K, iGpu UHD 630

Memory .. 32GB DDR4

Graphics card .. MSI RTX 2080 ti

Graphics driver .. latest studio

PSU .. Corsair 850i

Mboard .. Asus Z390 Code

 

Laptop ... (Acer Predator G9-793-77AC)

CPU .. i7-6700HQ Skylake-H

Memory ..32 GB DDR4, was previously 16 GB

Graphics card .. Nvidia GTX 1070

Graphics driver .. latest studio

Musicvid wrote on 6/30/2020, 6:10 AM

I just finished skimming this thread top to bottom, and I see I've got some catching up to do. I have been doing some tests independently, as I was challenged to repeat my banding tests from 2015 using the x265 8- and 10- bit encoders as a route to dithered 8-bit output, the question long being a favorite of internet hobbyists, but I digress.

The "averaging" cautions raised by @fifonik are especially appreciated; I suspect that paying close attention to RMSE regression data can be helpful here, but I avoided the issue (cheated?) by using static source.

I also got involved in a parallel discussion about the higher, and almost undifferentiated scores using x265's internal Tunes (the SSIM Tune being almost identical to ffmpeg). Using highest quality settings, the x265/ffmpeg numbers were maddeningly close to lossless, and as I said, undifferentiated. It struck me as being a virtual whitewash. See figure 2 below.

Part of today's tests also involved making a low resolution target to test against the venerable MSU QMT free utility, and with this static SD source, the MSU, x265 tune, and ffmpeg SSIM scores were almost identical. The HO QMT was about .01 lower, but as Wayne points out, is more differentiated, thus more useful for comparisons.

Figure 1.

The discussion about UT reminded me that my fascination with it began in 2011 as a result of visual difference comparisons, not numeric scores. Using the Belle Nuit target, it came in indistinguishable from Sony YUV, an acknowledged reference warhorse. So I'm going to be doing a lot more testing, one can be sure, in light of the inconsistencies being reported here. I'm still a huge fan of UT 422 as a lightweight alternative to ProRes.

For anyone interested, here are the initial static x265 tests, with the important enhanced visuals. I think that when it's a tossup between paradoxical data sets as described several times above, the inclusion of "some" visual data can help to settle the score (literally), and probably rightfully so.

This is especially true when PSNR (shadow noise) starts to become an issue, such as doing high quality restoration work with analog source, or trimming noise from Float -to- Integer projects.

Figure 2.

Being now the third hijacker to usurp @wwaag's uprezzing thread, I'm just going to sit back and be a fly on the wall ...

JN- wrote on 6/30/2020, 6:35 AM

Some of this reminds me, now for quite some time, of the old days and ways of Camera lens testing. It was always a challenge to interpret the sometimes byzantine comments of the reviewer, to see what was said and more importantly what wasn’t said in his conclusion and analysis of said lens.

Most importantly I always felt the reviewer would have one eye on the task in hand but both eyes on the lens manufacturers magazine adds contributions.

To a large extent thankfully, lens MTF testing removed this subjective only interpretation.

At least that way for sure you could eyeball the results yourself and agree or disagree with the reviewer, but having the maths was a great step forward in objectivity.

 

 

Last changed by JN- on 6/30/2020, 6:40 AM, changed a total of 1 times.

---------------------------------------------

Benchmarking thread

Codec Render Quality tables

---------------------------------------------

PC ... Corsair case, own build ...

CPU .. i9 9900K, iGpu UHD 630

Memory .. 32GB DDR4

Graphics card .. MSI RTX 2080 ti

Graphics driver .. latest studio

PSU .. Corsair 850i

Mboard .. Asus Z390 Code

 

Laptop ... (Acer Predator G9-793-77AC)

CPU .. i7-6700HQ Skylake-H

Memory ..32 GB DDR4, was previously 16 GB

Graphics card .. Nvidia GTX 1070

Graphics driver .. latest studio

Musicvid wrote on 6/30/2020, 7:10 AM

Yes, I remember those days very well.

 

wwaag wrote on 7/2/2020, 12:36 AM

I've just uploaded a new version of RQM FFmpeg that now computes VMAF as well as the other metrics. Here is a screen grab.

Note that you must use an FFmpeg version that includes the VMAF library. The latest Zeranoe nightly build of FFmpeg has that library and is what I've used for testing. https://ffmpeg.zeranoe.com/builds/

RQM FFmpeg can be downloaded in the HappyOtter free tools library at https://tools4vegas.com/library/