Tool for Measuring Render Quality

wwaag wrote on 1/6/2019, 11:39 AM

When assessing the quality of a render, most of us use the "looks OK to me" approach which is perfectly fine for our own stuff. However, when comparing quality across codecs, render settings, etc. a little more objectivity is desirable. A technique used in other threads (e.g. https://www.vegascreativesoftware.info/us/forum/cpu-render-vs-vce--114009/?page=1) has been to place the rendered content on a track above the original content and then set the compositing mode to Difference. While this is a definite improvement, it still doesn't offer an easy means of producing a summary estimate of overall render quality.

To this end, I have written a very simple app, Render Quality Metrics (RQM), which does a frame by frame comparison between the original and rendered footage, and produces two traditional measures of image quality: mean-squared-error (MSE) and the peak signal-to-noise ratio (PSNR). Here is a screenshot of the tool.

If anyone is interested, RQM, along with instructions for its use, may be downloaded here: https://tools4vegas.com/library/.

Many thanks to John Dennis and _JN who participated in the initial testing of this app. Hopefully, some of you may find the tool useful. Suggestions for improvement are welcome.

AKA the HappyOtter at https://tools4vegas.com/. System 1: Intel i7-8700k with HD 630 graphics plus an Nvidia RTX4070 graphics card. System 2: Intel i7-3770k with HD 4000 graphics plus an AMD RX550 graphics card. System 3: Laptop. Dell Inspiron Plus 16. Intel i7-11800H, Intel Graphics. Current cameras include Panasonic FZ2500, GoPro Hero11 and Hero8 Black plus a myriad of smartPhone, pocket cameras, video cameras and film cameras going back to the original Nikon S.

Comments

Musicvid wrote on 1/6/2019, 12:18 PM

Wonderful!

How does MSE and SSIM compare in function and application?

wwaag wrote on 1/6/2019, 2:34 PM

@Musicvid

If you do a search of ssim vs mse, you'll find lots of hits including the Wiki entry. https://en.wikipedia.org/wiki/Structural_similarity.

MSE is an easy to understand (also to implement) concept. For each frame, differences between the RGB components of each pixel are computed. The less the difference, the smaller error, and hence the greater similarity. For lossless codecs, all pixels will be the same. If you try a MagicYUV or UtVideo render comparison, you'll find MSE error to be 0 and PSNR to be 99, an arbitrary value since in its calculation, division by zero (MSE error) is mathematically impossible.

Quite frankly, SSIM is very complex mathematically and far beyond my pay grade. The MSU Video Quality Measurement Tool (http://www.compression.ru/video/quality_measure/vqmt_download.html) produces SSIM measures as well as other traditional measures, but $999 seems a bit pricey and the free download version has severe limitations.

Another reason for choosing MSE is that the computational method was already part of the SmartVideoTrim tool in HappyOtterScripts (https://tools4vegas.com/) and was thus easy to implement.

AKA the HappyOtter at https://tools4vegas.com/. System 1: Intel i7-8700k with HD 630 graphics plus an Nvidia RTX4070 graphics card. System 2: Intel i7-3770k with HD 4000 graphics plus an AMD RX550 graphics card. System 3: Laptop. Dell Inspiron Plus 16. Intel i7-11800H, Intel Graphics. Current cameras include Panasonic FZ2500, GoPro Hero11 and Hero8 Black plus a myriad of smartPhone, pocket cameras, video cameras and film cameras going back to the original Nikon S.

Former user wrote on 1/6/2019, 4:23 PM

I've updated the thread mentioned above using the new app, about half way down the page. https://www.vegascreativesoftware.info/us/forum/cpu-render-vs-vce--114009/?page=4

I put together some samples to give myself an idea of what's what in the order of things, for a varied type of scene, my idea of a general purpose test set.  If your curious about this, just a suggestion,  maybe put together a few scenes that you tend to use and check out the results. 

I had no VCE capability, (though the previously mentioned thread has) but I believe the result bears out the benefits of x264.  Anyway for my own limited needs I'll continue to use Nvenc for casual stuff.

This was a fairly high data rate set relative to what was tested in the previous thread.

The “Cpu” reference is simply Magix avc without hardware encoding.

 

Musicvid wrote on 1/6/2019, 5:30 PM

Yes, I have used the free (640x480) MSU SSIM tool here, by cropping sections from 1080 source.

https://www.vegascreativesoftware.info/us/forum/uncompressed-vs-sony-yuv-vs-magic-yuv-the-shootout--113650/#ca702514

It sounds like MSE is more along the lines of root mean square regression?

wwaag wrote on 1/6/2019, 6:10 PM

@Musicvid

Yes, it is essentially RMS error. My suspicion is that for comparing intermediates, the correlation between SSIM and MSE would be nearly 100%. Agree with your conclusion about MagicYUV--it's about as good as it gets--no loss and relatively good playback in Vegas.

AKA the HappyOtter at https://tools4vegas.com/. System 1: Intel i7-8700k with HD 630 graphics plus an Nvidia RTX4070 graphics card. System 2: Intel i7-3770k with HD 4000 graphics plus an AMD RX550 graphics card. System 3: Laptop. Dell Inspiron Plus 16. Intel i7-11800H, Intel Graphics. Current cameras include Panasonic FZ2500, GoPro Hero11 and Hero8 Black plus a myriad of smartPhone, pocket cameras, video cameras and film cameras going back to the original Nikon S.

Musicvid wrote on 1/6/2019, 6:36 PM

Except for perceptual weighting, MSE is probably fine for 99% of material.

PSNR doesn't seem to matter a lot with digital source. That's why 30-40% is considered good.

I could teach rms regression out of my head, ten years back.

For 422 Intermediates, UT comes in ahead of Magic if my numbers are correct.

Former user wrote on 1/7/2019, 11:12 AM

I've added the old Legacy MC and some intermediates.

Some updated tables ... https://www.dropbox.com/sh/xh8vkexgxif53m5/AAAH135JcFgIuqoQgFp5-o1za?dl=0

 

Former user wrote on 1/27/2019, 2:23 PM

My query is not specific to your free app, @wwaag thanks, its just that @john_dennis yourself and others have very recently being writing .bmp image sequences so rather than create a new post maybe try here with the problem.

I noticed quite early on that writing .bmp's to my PC appeared a lot slower than my laptop, if anything it should have been the other way around.

To do my testing with your app I invariably used the laptop where possible. With probably all of my testing done now I got around to checking out just how bad it is.

Writing for 60s creates just 18 frames on the desktop and 138 frames on the laptop. Of course this is from within VP directly and isn’t using the app, its a VP issue.

Rendering video clips on the PC is of course quicker than on the laptop, so I haven’t experienced any issues recently with the PC, it seems specific to writing .bmp's. It has me puzzled, any ideas?

I checked out tiff image sequence just now and theres no issue here, laptop 118 frames per minute, PC 148 frames per minute.

Former user wrote on 1/27/2019, 5:49 PM

Meant to also say that I disabled Anti Virus software on the PC, made no difference.

Former user wrote on 2/3/2019, 1:41 PM

Found a workaround by using a ramdrive on the PC, still no idea why only this image sequence is slow on the PC. I checked all of the other image sequences on laptop and PC, the PC was always faster to write.

Examples ...

using a 2 second test clip, 50 jpg images takes 5s on laptop, 3s on PC and 2s on Ramdrive

using a 2 second test clip, 50 bmp images takes 21s on laptop, 2m:53s on PC and 2s on Ramdrive.

fifonik wrote on 4/8/2019, 7:41 PM

Trying to use it and found that it is randomly stopped in the middle of processing on different frames.

No error message or anything else. It looks like Esc was pressed (but it was not).

Are there any logs/options available that might help to investigate what caused the issue?

bmp files were generated using bat file:

@echo off
mkdir %~n1
ffmpeg -i %1 %~n1/%%05d.bmp

However, not the images format causing the issue as the same files may be processed with no issues on the second or third run.

Thanks.

P.S. PSNR & SSIM can be calculated with ffmpeg as per manual (no images sequence generation required):

ffmpeg -i main.mp4 -i ref.mpg -lavfi "ssim;[0:v][1:v]psnr" -f null -

 

Last changed by fifonik on 4/8/2019, 11:24 PM, changed a total of 7 times.

Camcorder: Panasonic X1500 + Panasonic X920 + GoPro Hero 11 Black

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 5700X, RAM: G'Skill 16 GB DDR4@3200, Graphics card: MSI RX6600 8GB, SSD: Samsung 970 Evo+ 500GB (NVMe, OS), Samsung 870 Evo 1TB, HDD WD 4TB, HDD Toshiba 4TB, OS: Windows 10 Pro 22H2

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17, 18, 19, 22

Author of FFMetrics and FFBitrateViewer

john_dennis wrote on 4/8/2019, 9:35 PM

I downloaded a fresh copy of the Render Quality Measurement tool from the HOS site and ran the tests that I ran this morning from another system on the network. Same result as with the old RQM.

1) I'm running Windows 7 Pro on all systems.

2) All bitmaps were rendered from Vegas Pro of one version or other.

wwaag wrote on 4/8/2019, 10:15 PM

@fifonik

"Are there any logs/options available that might help to investigate what caused the issue?" If you tick "Save values for each frame", it should tell you the frame where it stopped. However, since it works on the next run, it's probably not the specific BMP that's causing the problem.

Thanks for posting the FFmpeg CL. Have you used it? Can the reference be an avi and the main be an MP4? If so, one could render uncompressed or lossless from Vegas and then compare it with a rendered mp4.

Last changed by wwaag on 4/8/2019, 10:16 PM, changed a total of 1 times.

AKA the HappyOtter at https://tools4vegas.com/. System 1: Intel i7-8700k with HD 630 graphics plus an Nvidia RTX4070 graphics card. System 2: Intel i7-3770k with HD 4000 graphics plus an AMD RX550 graphics card. System 3: Laptop. Dell Inspiron Plus 16. Intel i7-11800H, Intel Graphics. Current cameras include Panasonic FZ2500, GoPro Hero11 and Hero8 Black plus a myriad of smartPhone, pocket cameras, video cameras and film cameras going back to the original Nikon S.

fifonik wrote on 4/8/2019, 10:25 PM

The frame number stop increasing and Exit/Start/Reset buttons are enabled when this happen so I know on what frame it is stopped this time. But frame is different all the times in my case (I tried it on one PC only). I also tried png (they are smaller, why you suggested bmp?) with the same issue.

I ran the ffmpeg SSIM/PSNR calculation out of curiosity to check how PSNR value is aligned with the RQM's PSNR value. Yes, references can be in different formats, I tried m2ts (my sources) vs mp4 (my target) with no issues. I expect any format that supported by ffmpeg would work.

P.S. RQM's windows is resizable. I think it would be nice to set minimum limits as for now it can be decreased to almost title %-)

Last changed by fifonik on 4/8/2019, 10:27 PM, changed a total of 1 times.

Camcorder: Panasonic X1500 + Panasonic X920 + GoPro Hero 11 Black

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 5700X, RAM: G'Skill 16 GB DDR4@3200, Graphics card: MSI RX6600 8GB, SSD: Samsung 970 Evo+ 500GB (NVMe, OS), Samsung 870 Evo 1TB, HDD WD 4TB, HDD Toshiba 4TB, OS: Windows 10 Pro 22H2

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17, 18, 19, 22

Author of FFMetrics and FFBitrateViewer

Musicvid wrote on 4/8/2019, 10:35 PM

IP.S. PSNR & SSIM can be calculated with ffmpeg as per manual (no bmp sequence generation required):

There are specific x264 / x265 Tunes for SSIM and PSNR that must be used when using these benchmarks in ffmpeg and libav.

These Tunes, which are not for production, effectively disable psychovisual enhancements in x264 / x265, those serving to improve quality, but which totally mess up the quality measurements, and would give inconsistent results.

wwaag wrote on 4/8/2019, 10:38 PM

"so I know on what frame it is stopped this time." You're absolutely right. No need to look at the log.

Why BMP? It's lossless, plus no conversion within the app. Other formats are converted to BMPs. In any case, it probably doesn't make any difference.

"I think it would be nice to set minimum limits as for now it can be decreased to almost title".

That's an easy change to make it completely fixed.

AKA the HappyOtter at https://tools4vegas.com/. System 1: Intel i7-8700k with HD 630 graphics plus an Nvidia RTX4070 graphics card. System 2: Intel i7-3770k with HD 4000 graphics plus an AMD RX550 graphics card. System 3: Laptop. Dell Inspiron Plus 16. Intel i7-11800H, Intel Graphics. Current cameras include Panasonic FZ2500, GoPro Hero11 and Hero8 Black plus a myriad of smartPhone, pocket cameras, video cameras and film cameras going back to the original Nikon S.

fifonik wrote on 4/8/2019, 10:57 PM

There are specific x264 / x265 Tunes for SSIM and PSNR that must be used when using these benchmarks in ffmpeg and libav.

The tunes you mentioned are for in-build x264/265 encoders so better SSIM/PSNR metrics will be shown while analyzing encoded files.

The command suggested does not do any encoding. It just decodes provided files and do the metrics calculations.

Camcorder: Panasonic X1500 + Panasonic X920 + GoPro Hero 11 Black

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 5700X, RAM: G'Skill 16 GB DDR4@3200, Graphics card: MSI RX6600 8GB, SSD: Samsung 970 Evo+ 500GB (NVMe, OS), Samsung 870 Evo 1TB, HDD WD 4TB, HDD Toshiba 4TB, OS: Windows 10 Pro 22H2

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17, 18, 19, 22

Author of FFMetrics and FFBitrateViewer

fifonik wrote on 4/8/2019, 10:59 PM

No need to look at the log.

It would be nice if program add into the log reason for finishing. 'Stopped by user', 'Last frame reached', 'File 06.bmp could not be opened', 'Exception ....', etc.

Last changed by fifonik on 4/8/2019, 11:02 PM, changed a total of 1 times.

Camcorder: Panasonic X1500 + Panasonic X920 + GoPro Hero 11 Black

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 5700X, RAM: G'Skill 16 GB DDR4@3200, Graphics card: MSI RX6600 8GB, SSD: Samsung 970 Evo+ 500GB (NVMe, OS), Samsung 870 Evo 1TB, HDD WD 4TB, HDD Toshiba 4TB, OS: Windows 10 Pro 22H2

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17, 18, 19, 22

Author of FFMetrics and FFBitrateViewer

fifonik wrote on 4/8/2019, 11:13 PM

Got it. It was actually stopped by user (me).

I started the process then switched to another window (tested with browser, total commander, Notepad++) and continue to do something different.

When I pressed Esc key twice in that another program, the RQM is stopped while it is still not in focus.

Last changed by fifonik on 4/8/2019, 11:16 PM, changed a total of 1 times.

Camcorder: Panasonic X1500 + Panasonic X920 + GoPro Hero 11 Black

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 5700X, RAM: G'Skill 16 GB DDR4@3200, Graphics card: MSI RX6600 8GB, SSD: Samsung 970 Evo+ 500GB (NVMe, OS), Samsung 870 Evo 1TB, HDD WD 4TB, HDD Toshiba 4TB, OS: Windows 10 Pro 22H2

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17, 18, 19, 22

Author of FFMetrics and FFBitrateViewer

wwaag wrote on 4/9/2019, 12:30 AM

"When I pressed Esc key twice in that another program, the RQM is stopped while it is still not in focus."

Good. At least I understand the problem. Make sure that Stop processing can only occur if app has focus. Probably best to have a dedicated "Stop" button in the UI rather than use of the Escape key. That would avoid such problems altogether.

AKA the HappyOtter at https://tools4vegas.com/. System 1: Intel i7-8700k with HD 630 graphics plus an Nvidia RTX4070 graphics card. System 2: Intel i7-3770k with HD 4000 graphics plus an AMD RX550 graphics card. System 3: Laptop. Dell Inspiron Plus 16. Intel i7-11800H, Intel Graphics. Current cameras include Panasonic FZ2500, GoPro Hero11 and Hero8 Black plus a myriad of smartPhone, pocket cameras, video cameras and film cameras going back to the original Nikon S.

Former user wrote on 4/9/2019, 5:53 AM

A beep at end of processing would be useful, also a progress readout.

fifonik wrote on 4/9/2019, 3:50 PM

I've also found that if there are less than 3 frames in each folders (both test and ref) than when RQM finished nothing is added into log and keys are still disabled (and Esc does not work in this case). Calculated values are shown in UI.

Camcorder: Panasonic X1500 + Panasonic X920 + GoPro Hero 11 Black

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 5700X, RAM: G'Skill 16 GB DDR4@3200, Graphics card: MSI RX6600 8GB, SSD: Samsung 970 Evo+ 500GB (NVMe, OS), Samsung 870 Evo 1TB, HDD WD 4TB, HDD Toshiba 4TB, OS: Windows 10 Pro 22H2

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17, 18, 19, 22

Author of FFMetrics and FFBitrateViewer

Musicvid wrote on 4/9/2019, 4:32 PM

The command suggested does not do any encoding. It just decodes provided files and do the metrics calculations.

I see that now, and it's a great improvement over the ways I had been running SSIM comparisons.

That said, I've come to realize that PSNR is the more important benchmark test of production encoders, since it directly reflects shadow noise from the encoding.

For nonscientific purposes, wwaag's mean^2 regression tool probably tells us about all we need to know about replicability.

john_dennis wrote on 4/9/2019, 4:55 PM

" ...3 frames in each folders..."

Is the intermission after one frame or after two frames of that movie?