@Musicvid

If you do a search of ssim vs mse, you'll find lots of hits including the Wiki entry. https://en.wikipedia.org/wiki/Structural_similarity.

MSE is an easy to understand (also to implement) concept. For each frame, differences between the RGB components of each pixel are computed. The less the difference, the smaller error, and hence the greater similarity. For lossless codecs, all pixels will be the same. If you try a MagicYUV or UtVideo render comparison, you'll find MSE error to be 0 and PSNR to be 99, an arbitrary value since in its calculation, division by zero (MSE error) is mathematically impossible.

Quite frankly, SSIM is very complex mathematically and far beyond my pay grade. The MSU Video Quality Measurement Tool (http://www.compression.ru/video/quality_measure/vqmt_download.html) produces SSIM measures as well as other traditional measures, but $999 seems a bit pricey and the free download version has severe limitations.

Another reason for choosing MSE is that the computational method was already part of the SmartVideoTrim tool in HappyOtterScripts (https://tools4vegas.com/) and was thus easy to implement.

I've updated the thread mentioned above using the new app, about half way down the page. https://www.vegascreativesoftware.info/us/forum/cpu-render-vs-vce--114009/?page=4

I put together some samples to give myself an idea of what's what in the order of things, for a varied type of scene, my idea of a general purpose test set.  If your curious about this, just a suggestion,  maybe put together a few scenes that you tend to use and check out the results. 

I had no VCE capability, (though the previously mentioned thread has) but I believe the result bears out the benefits of x264.  Anyway for my own limited needs I'll continue to use Nvenc for casual stuff.

This was a fairly high data rate set relative to what was tested in the previous thread.

The “Cpu” reference is simply Magix avc without hardware encoding.

Yes, I have used the free (640x480) MSU SSIM tool here, by cropping sections from 1080 source.

https://www.vegascreativesoftware.info/us/forum/uncompressed-vs-sony-yuv-vs-magic-yuv-the-shootout--113650/#ca702514

It sounds like MSE is more along the lines of root mean square regression?

@Musicvid

Yes, it is essentially RMS error. My suspicion is that for comparing intermediates, the correlation between SSIM and MSE would be nearly 100%. Agree with your conclusion about MagicYUV--it's about as good as it gets--no loss and relatively good playback in Vegas.

Except for perceptual weighting, MSE is probably fine for 99% of material.

PSNR doesn't seem to matter a lot with digital source. That's why 30-40% is considered good.

I could teach rms regression out of my head, ten years back.

For 422 Intermediates, UT comes in ahead of Magic if my numbers are correct.

I've added the old Legacy MC and some intermediates.

Some updated tables ... https://www.dropbox.com/sh/xh8vkexgxif53m5/AAAH135JcFgIuqoQgFp5-o1za?dl=0

My query is not specific to your free app, @wwaag thanks, its just that @john_dennis yourself and others have very recently being writing .bmp image sequences so rather than create a new post maybe try here with the problem.

I noticed quite early on that writing .bmp's to my PC appeared a lot slower than my laptop, if anything it should have been the other way around.

To do my testing with your app I invariably used the laptop where possible. With probably all of my testing done now I got around to checking out just how bad it is.

Writing for 60s creates just 18 frames on the desktop and 138 frames on the laptop. Of course this is from within VP directly and isn’t using the app, its a VP issue.

Rendering video clips on the PC is of course quicker than on the laptop, so I haven’t experienced any issues recently with the PC, it seems specific to writing .bmp's. It has me puzzled, any ideas?

I checked out tiff image sequence just now and theres no issue here, laptop 118 frames per minute, PC 148 frames per minute.

Meant to also say that I disabled Anti Virus software on the PC, made no difference.

Found a workaround by using a ramdrive on the PC, still no idea why only this image sequence is slow on the PC. I checked all of the other image sequences on laptop and PC, the PC was always faster to write.

Examples ...

using a 2 second test clip, 50 jpg images takes 5s on laptop, 3s on PC and 2s on Ramdrive

using a 2 second test clip, 50 bmp images takes 21s on laptop, 2m:53s on PC and 2s on Ramdrive.

Trying to use it and found that it is randomly stopped in the middle of processing on different frames.

No error message or anything else. It looks like Esc was pressed (but it was not).

Are there any logs/options available that might help to investigate what caused the issue?

bmp files were generated using bat file:

@echo off
mkdir %~n1
ffmpeg -i %1 %~n1/%%05d.bmp

However, not the images format causing the issue as the same files may be processed with no issues on the second or third run.

Thanks.

P.S. PSNR & SSIM can be calculated with ffmpeg as per manual (no images sequence generation required):

ffmpeg -i main.mp4 -i ref.mpg -lavfi "ssim;[0:v][1:v]psnr" -f null -

I downloaded a fresh copy of the Render Quality Measurement tool from the HOS site and ran the tests that I ran this morning from another system on the network. Same result as with the old RQM.

1) I'm running Windows 7 Pro on all systems.

2) All bitmaps were rendered from Vegas Pro of one version or other.

@fifonik

"Are there any logs/options available that might help to investigate what caused the issue?" If you tick "Save values for each frame", it should tell you the frame where it stopped. However, since it works on the next run, it's probably not the specific BMP that's causing the problem.

Thanks for posting the FFmpeg CL. Have you used it? Can the reference be an avi and the main be an MP4? If so, one could render uncompressed or lossless from Vegas and then compare it with a rendered mp4.

The frame number stop increasing and Exit/Start/Reset buttons are enabled when this happen so I know on what frame it is stopped this time. But frame is different all the times in my case (I tried it on one PC only). I also tried png (they are smaller, why you suggested bmp?) with the same issue.

I ran the ffmpeg SSIM/PSNR calculation out of curiosity to check how PSNR value is aligned with the RQM's PSNR value. Yes, references can be in different formats, I tried m2ts (my sources) vs mp4 (my target) with no issues. I expect any format that supported by ffmpeg would work.

P.S. RQM's windows is resizable. I think it would be nice to set minimum limits as for now it can be decreased to almost title %-)

IP.S. PSNR & SSIM can be calculated with ffmpeg as per manual (no bmp sequence generation required):

There are specific x264 / x265 Tunes for SSIM and PSNR that must be used when using these benchmarks in ffmpeg and libav.

These Tunes, which are not for production, effectively disable psychovisual enhancements in x264 / x265, those serving to improve quality, but which totally mess up the quality measurements, and would give inconsistent results.

"so I know on what frame it is stopped this time." You're absolutely right. No need to look at the log.

Why BMP? It's lossless, plus no conversion within the app. Other formats are converted to BMPs. In any case, it probably doesn't make any difference.

"I think it would be nice to set minimum limits as for now it can be decreased to almost title".

That's an easy change to make it completely fixed.

There are specific x264 / x265 Tunes for SSIM and PSNR that must be used when using these benchmarks in ffmpeg and libav.

The tunes you mentioned are for in-build x264/265 encoders so better SSIM/PSNR metrics will be shown while analyzing encoded files.

The command suggested does not do any encoding. It just decodes provided files and do the metrics calculations.

No need to look at the log.

It would be nice if program add into the log reason for finishing. 'Stopped by user', 'Last frame reached', 'File 06.bmp could not be opened', 'Exception ....', etc.

Got it. It was actually stopped by user (me).

I started the process then switched to another window (tested with browser, total commander, Notepad++) and continue to do something different.

When I pressed Esc key twice in that another program, the RQM is stopped while it is still not in focus.

"When I pressed Esc key twice in that another program, the RQM is stopped while it is still not in focus."

Good. At least I understand the problem. Make sure that Stop processing can only occur if app has focus. Probably best to have a dedicated "Stop" button in the UI rather than use of the Escape key. That would avoid such problems altogether.

A beep at end of processing would be useful, also a progress readout.

I've also found that if there are less than 3 frames in each folders (both test and ref) than when RQM finished nothing is added into log and keys are still disabled (and Esc does not work in this case). Calculated values are shown in UI.

The command suggested does not do any encoding. It just decodes provided files and do the metrics calculations.

I see that now, and it's a great improvement over the ways I had been running SSIM comparisons.

That said, I've come to realize that PSNR is the more important benchmark test of production encoders, since it directly reflects shadow noise from the encoding.

For nonscientific purposes, wwaag's mean^2 regression tool probably tells us about all we need to know about replicability.