Three frames is a very small sample size for a video quality test. I have a project that I use that produces 914 frames with representative scenes, crossfades, etc.

Intermissions give the audience a break if a part of a program reaches, say 216,000 frames or so.

Don’t worry. Even (especially) fluent English speakers don’t understand much of what I say.

@fifonik

"P.S. PSNR & SSIM can be calculated with ffmpeg as per manual (no images sequence generation required):

ffmpeg -i main.mp4 -i ref.mpg -lavfi "ssim;[0:v][1:v]psnr" -f null -"

I used the above ffmpeg command line, thanks fifonik, and inserted the results into the table that I had previously made using the HO util.  It follows it mostly, small margin of error, I've highlighted and underscored the values that varied from the HO ones using ffmpeg, really, the trend is very similar overall.  I also made a second table with ffmpeg generated values only.

I used the ffmpeg average output for PSNR and “All” for SSIM. The SSIM had two values, a less than 1 decimal value and a larger value, I used the larger value.

Three frames is a very small sample size for a video quality test.

Sure.

However, the test video length does not matter for math (it must work and even one frame samples make sense) and also it is important for programming (programs must not show errors or halts even in such strange cases).

The SSIM had two values, a less than 1 decimal value and a larger value, I used the larger value

I think for SSIM the value that below 1 should be used. It is more common and the meaning is very simple: 1 -- frames are equal, 0 -- completely different. Bigger value is dB representation calculated as 10 * log10(1 / (1 - ssim))

Given the problems with RQM, I've uploaded a new version with the following changes:

• Dialog is now fixed size
• Added Stop button rather than use of Escape key.
• Now works with all frame comparison sizes including 1, 2, 3. My system worked OK for 2 and 3 changes, but hung if just a single comparison was made. I added a bit of time delay upon completion of the frame comparisons and it now seems to work OK. If not, let me know.

The new version can be downloaded here. https://tools4vegas.com/library/

Thanks for these changes. It's a bit sad that the program does not accept PNGs any longer.

P.S. Let me know if you'd like to know about other tiny glitches in the program =)

Just uploaded another version. There is now a check for 0 images and it will work with other image formats. I tested with BMPs, PNGs, JPGs and TIFs. Results were the same for all except JPGs although fairly "close". The one advantage of using JPGs is that rendering times from Vegas are much quicker. Here are the results from a 1080 60 frame test. Same processing speed except for Tiffs which were slower.

Date: 2019/04/09  19:49:25  
Description: Butterfly_BMPs
Frames Processed: 60
Processing Speed: 5.45 fps
Mean Squared Error: 10.898
Peak Signal to Noise Ratio: 37.828

Date: 2019/04/09  20:14:04  
Description: Butterfly_PNGs
Frames Processed: 60
Processing Speed: 5.45 fps
Mean Squared Error: 10.898
Peak Signal to Noise Ratio: 37.828

Date: 2019/04/09  20:15:48  
Description: Butterfly_JPGs
Frames Processed: 60
Processing Speed: 5.45 fps
Mean Squared Error: 10.982
Peak Signal to Noise Ratio: 37.775

Date: 2019/04/09  20:18:06  
Description: Butterfly_TIFfs
Frames Processed: 60
Processing Speed: 4 fps
Mean Squared Error: 10.898
Peak Signal to Noise Ratio: 37.828

@fifonik

"Let me know if you'd like to know about other tiny glitches in the program"

Sure--tiny glitches are better than major glitches.

Thanks for bringing back PNGs support. I prefer them over BMP/TIFF as they are smaller and also lossless.

Here you are:

1. Start the process and while it is going put mouse pointer over Stop button => 'Begin the comparison button'

2. Put mouse pointer over bottom right corner of the program window => cursor changes to resizing one, but the resize is not available any longer.

3. Put mouse pointer over Rest button => 'Reset the description and results textboxes'. But it does not reset description, only results.

4. Esc => program closes. I think, as calculation is long process it should not do this without confirmation. I would not do it at all for the program. Exit button and Alt + F4 should be more than enough.

5. I'd change titles (if possible) for locate folder dialogs. When distracted I might not remember what am I selecting: test or ref. In addition, I'd remove 'Make new folder' button from these dialogs as it does not make much sense.

6. It'd be nice to see equal vertical distances between folder input fields. My CDO drives me crazy when I see this %-)

Sorry. Feel free to ignore all this...

The SSIM had two values, a less than 1 decimal value and a larger value, I used the larger value

I think for SSIM the value that below 1 should be used. It is more common and the meaning is very simple: 1 -- frames are equal, 0 -- completely different. Bigger value is dB representation calculated as 10 * log10(1 / (1 - ssim))

@fifonik
The table on the right now has the SSIM column with values below 1.  Result is similar.

The original/test files are 27s duration, 675 frames.

Below is an Intermediate file multi generation comparison, I’ve also included a MagicYUV lossless sample.

Just uploaded a new version with mostly cosmetic changes.

@fifonik

I've made most of the changes you suggested. #1 Fixed. #2 is something that cannot be changed (AFAIK). #3 Changed. #4 had already been changed--the Escape key no longer works in RQM. #5 Like #2, these dialogs cannot be changed since they are part of Windows. #6 Fixed--don't want to drive anyone crazy--LOL. As always, thanks for your comments.

Posted this elsewhere:

but, Why does x264 AVC CPU render show up with numbers like that?? is that normal? (I've done this test twice thinking something has gone wrong..)

@AVsupport

All of your numbers are unreasonably high for some reason. Can you upload a short sample somewhere that could be downloaded?

I've uploaded 5 original and 5 AVC CPU renders on my google, but this was a complete series of 528. Not sure what values one normally should have expected @wwaag ? AAAhhhh I see looking at those images, I believe I made a mistake...there's some with no FX applied!! That old chestnut!! Sorry everybody, stupid me

anyway, files are here: https://drive.google.com/open?id=1fQsFXa2bASSYFo6fgf27QOxQjv-OGd4H

Here's the updated spreadsheet. Please disregard first lot of test since they were comparing the original with intermediates with FA applied. Sorry for the confusion folks. @wwaag Does that now look a little more like it?

I'm just getting an error message "Unequal number of frames" when I run this. I tried it first with 272 BMPs in each folder. Then I tried it with just 6. I've tried moving the folders to different drives. Any ideas?

@NickHope Make certain that the "Paths" are correct ...

I understand they are stored in the registry. Change them both i.e. Test folder and original folder.

On my PC, not on my laptop, weird, testing BMP's was much slower, so I used TIFF, much faster. @wwaag confirmed previously was ok.

@JN- I've double-checked the paths. Also tried moving and changing the folders again. No joy I'm afraid. I then found the entries in the registry and saw that they were still at the original defaults. So I reset my folder paths to match those, but I'm still getting the error.

@NickHope I just now remembered, in case you haven’t, you must put a value in the description.

I have written a new version of RQM which includes a script for a direct render from Vegas using the Frameserver which is automatically configured for writing BMP's.

There is also an option to compute SSIM in addition to the usual MSE and PSNR. It is much faster and doesn't require two sets of images to be compared. It makes use of setting the rendered track Compositing Mode to Difference. So basically just place the render file on the top track, select a loop region for test, and run the script. There's also an option to select every nth frame which really speeds things up and yields essentially the same results. Here's a demo.

If there's any interest, I'll try to upload these to the HOS Free Tools library sooner rather than later. It's another project that's been on the "back burner".

@wwaag Is it a standalone tool or does it need your full program to be installed and used? The SSIM is a nice addition.

It's a standalone--just like the original version--no need to install HOS. It makes use of the same registry locations as the original version.