Tool for Measuring Render Quality

Comments

fifonik wrote on 4/9/2019, 5:17 PM

Not sure what exactly do you mean (sorry, English is my 2nd language).

There is no intermission. Looks like it is calculated as it should. But the GUI is not recovered (buttons still disabled) and I cannot run it again without restarting program (it was '...less then 3...', it works fine with 3 frames, does not with 2, 1 or 0 frames )

As PSNR value generated by the tool not matched to ffmpeg's generated value I tried to check single frame's value and how average is calculated (easier to do with 2 frames than 200).

Last changed by fifonik on 4/9/2019, 5:21 PM, changed a total of 2 times.

Camcorder: Panasonic X920

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 5700X, RAM: G'Skill 16 GB DDR4@3200, Graphics card: MSI RX6600 8GB, SSD: Samsung 970 Evo+ 1TB (NVMe, OS), Samsung 870 Evo, HDD Seagate 3TB, HDD Toshiba 4TB, OS: Windows 10 Pro 22H2

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17, 18, 19

john_dennis wrote on 4/9/2019, 5:50 PM

Three frames is a very small sample size for a video quality test. I have a project that I use that produces 914 frames with representative scenes, crossfades, etc.

Intermissions give the audience a break if a part of a program reaches, say 216,000 frames or so.

Don’t worry. Even (especially) fluent English speakers don’t understand much of what I say.

Last changed by john_dennis on 4/9/2019, 5:55 PM, changed a total of 1 times.

My main system:
Motherboard: ASUS ProArt Z790-CREATOR WIFI
CPU: Intel Core i9-13900K - Core i9 13th Gen Raptor Lake 24-Core (8P+16E) P-core Base Frequency: 3.0 GHz E-core Base Frequency: 2.2 GHz LGA 1700 125W Intel UHD Graphics 770 Desktop Processor - BX8071513900K
GPU: Currently intel on-die video adapter
RAM: CORSAIR Vengeance 64GB (2 x 32GB) 288-Pin PC RAM DDR5 5600 (PC5 44800) Desktop Memory Model CMK64GX5M2B5600C40
Disk O/S & Programs: WD Black SN850 NVMe SSD WDS100T1X0E - SSD - 1 TB - PCIe 4.0 x4 (NVMe)
Disk Active Projects: 1TB & 2TB WD BLACK SN750 NVMe Internal PCI Express 3.0 x4 Solid State Drives
Disk Other: WD Ultrastar/Hitachi Hard Drives: WDBBUR0080BNC-WRSN, HGST HUH728080ALE600, 724040ALE640, HDS3020BLA642
Case: LIAN LI PC-90 Black Aluminum ATX Full Tower Case
CPU cooling: CORSAIR - iCUE H115i RGB PRO XT 280mm Radiator CPU Liquid Cooling System
Power supply: SeaSonic SS-750KM3 750W 80 PLUS GOLD Certified Full Modular Active PFC Power Supply
Drive Bay: Kingwin KF-256-BK 2.5" and 3.5" Trayless Hot Swap Rack with USB 3
Sound card: Realtek S1220A on motherboard. Recording done on another system.
Primary Monitor: ASUS ProArt 31.5" 1440p HDR10 Monitor PA328QV
O/S: Windows 10 Pro 10.0.19045 Build 19045
Camera: Sony RX10 Model IV

https://www.youtube.com/user/thedennischannel

Former user wrote on 4/9/2019, 6:17 PM

@fifonik

"P.S. PSNR & SSIM can be calculated with ffmpeg as per manual (no images sequence generation required):

ffmpeg -i main.mp4 -i ref.mpg -lavfi "ssim;[0:v][1:v]psnr" -f null -"

 

I used the above ffmpeg command line, thanks fifonik, and inserted the results into the table that I had previously made using the HO util.  It follows it mostly, small margin of error, I've highlighted and underscored the values that varied from the HO ones using ffmpeg, really, the trend is very similar overall.  I also made a second table with ffmpeg generated values only.

I used the ffmpeg average output for PSNR and “All” for SSIM. The SSIM had two values, a less than 1 decimal value and a larger value, I used the larger value.

fifonik wrote on 4/9/2019, 6:59 PM

Three frames is a very small sample size for a video quality test.

Sure.

However, the test video length does not matter for math (it must work and even one frame samples make sense) and also it is important for programming (programs must not show errors or halts even in such strange cases).

Camcorder: Panasonic X920

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 5700X, RAM: G'Skill 16 GB DDR4@3200, Graphics card: MSI RX6600 8GB, SSD: Samsung 970 Evo+ 1TB (NVMe, OS), Samsung 870 Evo, HDD Seagate 3TB, HDD Toshiba 4TB, OS: Windows 10 Pro 22H2

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17, 18, 19

fifonik wrote on 4/9/2019, 7:06 PM
The SSIM had two values, a less than 1 decimal value and a larger value, I used the larger value

I think for SSIM the value that below 1 should be used. It is more common and the meaning is very simple: 1 -- frames are equal, 0 -- completely different. Bigger value is dB representation calculated as 10 * log10(1 / (1 - ssim))

Camcorder: Panasonic X920

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 5700X, RAM: G'Skill 16 GB DDR4@3200, Graphics card: MSI RX6600 8GB, SSD: Samsung 970 Evo+ 1TB (NVMe, OS), Samsung 870 Evo, HDD Seagate 3TB, HDD Toshiba 4TB, OS: Windows 10 Pro 22H2

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17, 18, 19

wwaag wrote on 4/9/2019, 7:41 PM

Given the problems with RQM, I've uploaded a new version with the following changes:

  • Dialog is now fixed size
  • Added Stop button rather than use of Escape key.
  • Now works with all frame comparison sizes including 1, 2, 3. My system worked OK for 2 and 3 changes, but hung if just a single comparison was made. I added a bit of time delay upon completion of the frame comparisons and it now seems to work OK. If not, let me know.

The new version can be downloaded here. https://tools4vegas.com/library/

AKA the HappyOtter at https://tools4vegas.com/. System 1: Intel i7-8700k with HD 630 graphics plus an Nvidia 1050ti graphics card. System 2: Intel i7-3770k with HD 4000 graphics plus an AMD RX550 graphics card. System 3: Laptop. Dell Inspiron Plus 16. Intel i7-11800H, Intel Graphics. Current cameras include Panasonic FZ2500, GoPro Hero11 and Hero8 Black plus a myriad of smartPhone, pocket cameras, video cameras and film cameras going back to the original Nikon S.

fifonik wrote on 4/9/2019, 8:15 PM

Thanks for these changes. It's a bit sad that the program does not accept PNGs any longer.

P.S. Let me know if you'd like to know about other tiny glitches in the program =)

Camcorder: Panasonic X920

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 5700X, RAM: G'Skill 16 GB DDR4@3200, Graphics card: MSI RX6600 8GB, SSD: Samsung 970 Evo+ 1TB (NVMe, OS), Samsung 870 Evo, HDD Seagate 3TB, HDD Toshiba 4TB, OS: Windows 10 Pro 22H2

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17, 18, 19

wwaag wrote on 4/9/2019, 10:30 PM

Just uploaded another version. There is now a check for 0 images and it will work with other image formats. I tested with BMPs, PNGs, JPGs and TIFs. Results were the same for all except JPGs although fairly "close". The one advantage of using JPGs is that rendering times from Vegas are much quicker. Here are the results from a 1080 60 frame test. Same processing speed except for Tiffs which were slower.

Date: 2019/04/09  19:49:25  
Description: Butterfly_BMPs
Frames Processed: 60
Processing Speed: 5.45 fps
Mean Squared Error: 10.898
Peak Signal to Noise Ratio: 37.828

Date: 2019/04/09  20:14:04  
Description: Butterfly_PNGs
Frames Processed: 60
Processing Speed: 5.45 fps
Mean Squared Error: 10.898
Peak Signal to Noise Ratio: 37.828

Date: 2019/04/09  20:15:48  
Description: Butterfly_JPGs
Frames Processed: 60
Processing Speed: 5.45 fps
Mean Squared Error: 10.982
Peak Signal to Noise Ratio: 37.775

Date: 2019/04/09  20:18:06  
Description: Butterfly_TIFfs
Frames Processed: 60
Processing Speed: 4 fps
Mean Squared Error: 10.898
Peak Signal to Noise Ratio: 37.828

 

AKA the HappyOtter at https://tools4vegas.com/. System 1: Intel i7-8700k with HD 630 graphics plus an Nvidia 1050ti graphics card. System 2: Intel i7-3770k with HD 4000 graphics plus an AMD RX550 graphics card. System 3: Laptop. Dell Inspiron Plus 16. Intel i7-11800H, Intel Graphics. Current cameras include Panasonic FZ2500, GoPro Hero11 and Hero8 Black plus a myriad of smartPhone, pocket cameras, video cameras and film cameras going back to the original Nikon S.

wwaag wrote on 4/9/2019, 10:33 PM

@fifonik

"Let me know if you'd like to know about other tiny glitches in the program"

Sure--tiny glitches are better than major glitches.

AKA the HappyOtter at https://tools4vegas.com/. System 1: Intel i7-8700k with HD 630 graphics plus an Nvidia 1050ti graphics card. System 2: Intel i7-3770k with HD 4000 graphics plus an AMD RX550 graphics card. System 3: Laptop. Dell Inspiron Plus 16. Intel i7-11800H, Intel Graphics. Current cameras include Panasonic FZ2500, GoPro Hero11 and Hero8 Black plus a myriad of smartPhone, pocket cameras, video cameras and film cameras going back to the original Nikon S.

fifonik wrote on 4/9/2019, 11:22 PM

Thanks for bringing back PNGs support. I prefer them over BMP/TIFF as they are smaller and also lossless.

Here you are:

1. Start the process and while it is going put mouse pointer over Stop button => 'Begin the comparison button'

2. Put mouse pointer over bottom right corner of the program window => cursor changes to resizing one, but the resize is not available any longer.

3. Put mouse pointer over Rest button => 'Reset the description and results textboxes'. But it does not reset description, only results.

4. Esc => program closes. I think, as calculation is long process it should not do this without confirmation. I would not do it at all for the program. Exit button and Alt + F4 should be more than enough.

5. I'd change titles (if possible) for locate folder dialogs. When distracted I might not remember what am I selecting: test or ref. In addition, I'd remove 'Make new folder' button from these dialogs as it does not make much sense.

6. It'd be nice to see equal vertical distances between folder input fields. My CDO drives me crazy when I see this %-)

Sorry. Feel free to ignore all this...

Camcorder: Panasonic X920

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 5700X, RAM: G'Skill 16 GB DDR4@3200, Graphics card: MSI RX6600 8GB, SSD: Samsung 970 Evo+ 1TB (NVMe, OS), Samsung 870 Evo, HDD Seagate 3TB, HDD Toshiba 4TB, OS: Windows 10 Pro 22H2

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17, 18, 19

Former user wrote on 4/10/2019, 10:00 AM
The SSIM had two values, a less than 1 decimal value and a larger value, I used the larger value

I think for SSIM the value that below 1 should be used. It is more common and the meaning is very simple: 1 -- frames are equal, 0 -- completely different. Bigger value is dB representation calculated as 10 * log10(1 / (1 - ssim))

@fifonik
The table on the right now has the SSIM column with values below 1.  Result is similar.

The original/test files are 27s duration, 675 frames.

 

Below is an Intermediate file multi generation comparison, I’ve also included a MagicYUV lossless sample.

 

 

 

wwaag wrote on 4/10/2019, 11:40 AM

Just uploaded a new version with mostly cosmetic changes.

@fifonik

I've made most of the changes you suggested. #1 Fixed. #2 is something that cannot be changed (AFAIK). #3 Changed. #4 had already been changed--the Escape key no longer works in RQM. #5 Like #2, these dialogs cannot be changed since they are part of Windows. #6 Fixed--don't want to drive anyone crazy--LOL. As always, thanks for your comments.

AKA the HappyOtter at https://tools4vegas.com/. System 1: Intel i7-8700k with HD 630 graphics plus an Nvidia 1050ti graphics card. System 2: Intel i7-3770k with HD 4000 graphics plus an AMD RX550 graphics card. System 3: Laptop. Dell Inspiron Plus 16. Intel i7-11800H, Intel Graphics. Current cameras include Panasonic FZ2500, GoPro Hero11 and Hero8 Black plus a myriad of smartPhone, pocket cameras, video cameras and film cameras going back to the original Nikon S.

AVsupport wrote on 4/17/2019, 2:14 AM

Posted this elsewhere:

but, Why does x264 AVC CPU render show up with numbers like that?? is that normal? (I've done this test twice thinking something has gone wrong..)

my current Win10/64 system (latest drivers, water cooled) :

Intel Coffee Lake i5 Hexacore (unlocked, but not overclocked) 4.0 GHz on Z370 chipset board,

32GB (4x8GB Corsair Dual Channel DDR4-2133) XMP-3000 RAM,

Intel 600series 512GB M.2 SSD system drive running Win10/64 home automatic driver updates,

Crucial BX500 1TB EDIT 3D NAND SATA 2.5-inch SSD

2x 4TB 7200RPM NAS HGST data drive,

Intel HD630 iGPU - currently disabled in Bios,

nVidia GTX1060 6GB, always on latest [creator] drivers. nVidia HW acceleration enabled.

main screen 4K/50p 1ms scaled @175%, second screen 1920x1080/50p 1ms.

wwaag wrote on 4/17/2019, 8:14 PM

@AVsupport

All of your numbers are unreasonably high for some reason. Can you upload a short sample somewhere that could be downloaded?

AKA the HappyOtter at https://tools4vegas.com/. System 1: Intel i7-8700k with HD 630 graphics plus an Nvidia 1050ti graphics card. System 2: Intel i7-3770k with HD 4000 graphics plus an AMD RX550 graphics card. System 3: Laptop. Dell Inspiron Plus 16. Intel i7-11800H, Intel Graphics. Current cameras include Panasonic FZ2500, GoPro Hero11 and Hero8 Black plus a myriad of smartPhone, pocket cameras, video cameras and film cameras going back to the original Nikon S.

AVsupport wrote on 4/17/2019, 9:26 PM

I've uploaded 5 original and 5 AVC CPU renders on my google, but this was a complete series of 528. Not sure what values one normally should have expected @wwaag ? AAAhhhh I see looking at those images, I believe I made a mistake...there's some with no FX applied!! That old chestnut!! Sorry everybody, stupid me

anyway, files are here: https://drive.google.com/open?id=1fQsFXa2bASSYFo6fgf27QOxQjv-OGd4H

Last changed by AVsupport on 4/18/2019, 12:23 AM, changed a total of 3 times.

my current Win10/64 system (latest drivers, water cooled) :

Intel Coffee Lake i5 Hexacore (unlocked, but not overclocked) 4.0 GHz on Z370 chipset board,

32GB (4x8GB Corsair Dual Channel DDR4-2133) XMP-3000 RAM,

Intel 600series 512GB M.2 SSD system drive running Win10/64 home automatic driver updates,

Crucial BX500 1TB EDIT 3D NAND SATA 2.5-inch SSD

2x 4TB 7200RPM NAS HGST data drive,

Intel HD630 iGPU - currently disabled in Bios,

nVidia GTX1060 6GB, always on latest [creator] drivers. nVidia HW acceleration enabled.

main screen 4K/50p 1ms scaled @175%, second screen 1920x1080/50p 1ms.

AVsupport wrote on 4/18/2019, 4:05 AM

Here's the updated spreadsheet. Please disregard first lot of test since they were comparing the original with intermediates with FA applied. Sorry for the confusion folks. @wwaag Does that now look a little more like it?

 

my current Win10/64 system (latest drivers, water cooled) :

Intel Coffee Lake i5 Hexacore (unlocked, but not overclocked) 4.0 GHz on Z370 chipset board,

32GB (4x8GB Corsair Dual Channel DDR4-2133) XMP-3000 RAM,

Intel 600series 512GB M.2 SSD system drive running Win10/64 home automatic driver updates,

Crucial BX500 1TB EDIT 3D NAND SATA 2.5-inch SSD

2x 4TB 7200RPM NAS HGST data drive,

Intel HD630 iGPU - currently disabled in Bios,

nVidia GTX1060 6GB, always on latest [creator] drivers. nVidia HW acceleration enabled.

main screen 4K/50p 1ms scaled @175%, second screen 1920x1080/50p 1ms.

NickHope wrote on 3/10/2020, 7:41 AM

I'm just getting an error message "Unequal number of frames" when I run this. I tried it first with 272 BMPs in each folder. Then I tried it with just 6. I've tried moving the folders to different drives. Any ideas?

JN- wrote on 3/10/2020, 8:08 AM

@NickHope Make certain that the "Paths" are correct ...

I understand they are stored in the registry. Change them both i.e. Test folder and original folder.

On my PC, not on my laptop, weird, testing BMP's was much slower, so I used TIFF, much faster. @wwaag confirmed previously was ok.

Last changed by JN- on 3/10/2020, 8:12 AM, changed a total of 1 times.

---------------------------------------------

VFR2CFR, Variable frame rate to Constant frame rate link to zip here.

Copies Video Converts Audio to AAC, link to zip here.

Convert 2 Lossless, link to ZIP here.

Convert Odd 2 Even (frame size), link to ZIP here

Benchmarking Continued thread + link to zip here

Codec Render Quality tables zip

---------------------------------------------

PC ... Corsair case, own build ...

CPU .. i9 9900K, iGpu UHD 630

Memory .. 32GB DDR4

Graphics card .. MSI RTX 2080 ti

Graphics driver .. latest studio

PSU .. Corsair 850i

Mboard .. Asus Z390 Code

 

Laptop… XMG

i9-11900k, iGpu n/a

Memory 64GB DDR4

Graphics card … Laptop RTX 3080

NickHope wrote on 3/10/2020, 9:22 AM

@JN- I've double-checked the paths. Also tried moving and changing the folders again. No joy I'm afraid. I then found the entries in the registry and saw that they were still at the original defaults. So I reset my folder paths to match those, but I'm still getting the error.

JN- wrote on 3/10/2020, 9:41 AM

@NickHope I just now remembered, in case you haven’t, you must put a value in the description.

Last changed by JN- on 3/10/2020, 9:41 AM, changed a total of 1 times.

---------------------------------------------

VFR2CFR, Variable frame rate to Constant frame rate link to zip here.

Copies Video Converts Audio to AAC, link to zip here.

Convert 2 Lossless, link to ZIP here.

Convert Odd 2 Even (frame size), link to ZIP here

Benchmarking Continued thread + link to zip here

Codec Render Quality tables zip

---------------------------------------------

PC ... Corsair case, own build ...

CPU .. i9 9900K, iGpu UHD 630

Memory .. 32GB DDR4

Graphics card .. MSI RTX 2080 ti

Graphics driver .. latest studio

PSU .. Corsair 850i

Mboard .. Asus Z390 Code

 

Laptop… XMG

i9-11900k, iGpu n/a

Memory 64GB DDR4

Graphics card … Laptop RTX 3080

wwaag wrote on 3/10/2020, 12:06 PM

I have written a new version of RQM which includes a script for a direct render from Vegas using the Frameserver which is automatically configured for writing BMP's.

There is also an option to compute SSIM in addition to the usual MSE and PSNR. It is much faster and doesn't require two sets of images to be compared. It makes use of setting the rendered track Compositing Mode to Difference. So basically just place the render file on the top track, select a loop region for test, and run the script. There's also an option to select every nth frame which really speeds things up and yields essentially the same results. Here's a demo.

If there's any interest, I'll try to upload these to the HOS Free Tools library sooner rather than later. It's another project that's been on the "back burner".

 

AKA the HappyOtter at https://tools4vegas.com/. System 1: Intel i7-8700k with HD 630 graphics plus an Nvidia 1050ti graphics card. System 2: Intel i7-3770k with HD 4000 graphics plus an AMD RX550 graphics card. System 3: Laptop. Dell Inspiron Plus 16. Intel i7-11800H, Intel Graphics. Current cameras include Panasonic FZ2500, GoPro Hero11 and Hero8 Black plus a myriad of smartPhone, pocket cameras, video cameras and film cameras going back to the original Nikon S.

JN- wrote on 3/10/2020, 12:36 PM

@wwaag Is it a standalone tool or does it need your full program to be installed and used? The SSIM is a nice addition.

Last changed by JN- on 3/10/2020, 12:36 PM, changed a total of 1 times.

---------------------------------------------

VFR2CFR, Variable frame rate to Constant frame rate link to zip here.

Copies Video Converts Audio to AAC, link to zip here.

Convert 2 Lossless, link to ZIP here.

Convert Odd 2 Even (frame size), link to ZIP here

Benchmarking Continued thread + link to zip here

Codec Render Quality tables zip

---------------------------------------------

PC ... Corsair case, own build ...

CPU .. i9 9900K, iGpu UHD 630

Memory .. 32GB DDR4

Graphics card .. MSI RTX 2080 ti

Graphics driver .. latest studio

PSU .. Corsair 850i

Mboard .. Asus Z390 Code

 

Laptop… XMG

i9-11900k, iGpu n/a

Memory 64GB DDR4

Graphics card … Laptop RTX 3080

wwaag wrote on 3/10/2020, 12:55 PM

It's a standalone--just like the original version--no need to install HOS. It makes use of the same registry locations as the original version.

AKA the HappyOtter at https://tools4vegas.com/. System 1: Intel i7-8700k with HD 630 graphics plus an Nvidia 1050ti graphics card. System 2: Intel i7-3770k with HD 4000 graphics plus an AMD RX550 graphics card. System 3: Laptop. Dell Inspiron Plus 16. Intel i7-11800H, Intel Graphics. Current cameras include Panasonic FZ2500, GoPro Hero11 and Hero8 Black plus a myriad of smartPhone, pocket cameras, video cameras and film cameras going back to the original Nikon S.

JN- wrote on 3/10/2020, 2:39 PM

@wwaag 👍 Whens it’s available ill redo some of my tests and add in your SSIM.

Last changed by JN- on 3/10/2020, 2:41 PM, changed a total of 1 times.

---------------------------------------------

VFR2CFR, Variable frame rate to Constant frame rate link to zip here.

Copies Video Converts Audio to AAC, link to zip here.

Convert 2 Lossless, link to ZIP here.

Convert Odd 2 Even (frame size), link to ZIP here

Benchmarking Continued thread + link to zip here

Codec Render Quality tables zip

---------------------------------------------

PC ... Corsair case, own build ...

CPU .. i9 9900K, iGpu UHD 630

Memory .. 32GB DDR4

Graphics card .. MSI RTX 2080 ti

Graphics driver .. latest studio

PSU .. Corsair 850i

Mboard .. Asus Z390 Code

 

Laptop… XMG

i9-11900k, iGpu n/a

Memory 64GB DDR4

Graphics card … Laptop RTX 3080