Tool for Measuring Render Quality - Version 2

wwaag wrote on 3/18/2020, 3:35 PM

Over a year ago, I started a thread https://www.vegascreativesoftware.info/us/forum/tool-for-measuring-render-quality--114398/?page=1 which described a tool, Render Quality Metrics (RQM) for measuring the quality of renders from Vegas. Admittedly, RQM was pretty "clunky" in that it required the user to render an image sequence of BMPs for both the Original and Rendered files. A new version is available in the Happy Otter Free Tools Library which greatly simplifies and speeds up its use. In addition to Mean Squared Error (MSE) and Power Signal to Noise Ratio (PSNR), RQM now has an option to compute the Structural Similarity Index Metric (SSIM) although it is very slow. RQM now has two apps, a script that renders the image sequence from Vegas and a standalone app that does the computation.

Rather than render two sets of BMPs, RQM makes use of the track compositing mode in Vegas. The rendered file is placed on the top track and its compositing mode is set to "Difference". Another difference is that the BMPs are now directly rendered using the DebugMode FrameServer (DMFS) which speeds up rendering. I've done testing to make sure that BMPs from DMFS are identical to those produced by Vegas. I've also done testing to make sure that the single BMP based on the composite difference in Vegas is the same as the difference from separately rendered BMPs.

There are also options to "speed up" the process by selecting the pixel block size during creation of the BMPs and also an option to select every nth frame during processing. Here are screen grabs of both the script and standalone app.

A demo of its usage can be found at https://vimeo.com/398614859

The new version is titled Vegas Render Quality and can be downloaded at https://tools4vegas.com/library/ The zip file contains both the script and the standalone app.

Comments and suggestions for improvement are welcome.

Comments

Musicvid wrote on 3/18/2020, 8:09 PM

I have put the .dll and .png in the path shown. It does nothing when I click it.

Also I put it in Vegas Script Menu.

System.IO.FileLoadException: Could not load file or assembly 'file:///C:\Program Files\VEGAS\VEGAS Pro 14.0\Script Menu\Happy Otter Scripts\RenderQuality.dll' or one of its dependencies. Operation is not supported. (Exception from HRESULT: 0x80131515)
File name: 'file:///C:\Program Files\VEGAS\VEGAS Pro 14.0\Script Menu\Happy Otter Scripts\RenderQuality.dll' ---> System.NotSupportedException: An attempt was made to load an assembly from a network location which would have caused the assembly to be sandboxed in previous versions of the .NET Framework. This release of the .NET Framework does not enable CAS policy by default, so this load may be dangerous. If this load is not intended to sandbox the assembly, please enable the loadFromRemoteSources switch. See http://go.microsoft.com/fwlink/?LinkId=155569 for more information.
   at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, RuntimeAssembly reqAssembly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.RuntimeAssembly.InternalLoadFrom(String assemblyFile, Evidence securityEvidence, Byte[] hashValue, AssemblyHashAlgorithm hashAlgorithm, Boolean forIntrospection, Boolean suppressSecurityChecks, StackCrawlMark& stackMark)
   at System.Reflection.Assembly.LoadFrom(String assemblyFile)
   at ScriptPortal.Vegas.ScriptHost.PrecompiledScriptManager.Run()
   at ScriptPortal.Vegas.ScriptHost.RunScript(Boolean fCompileOnly)

I also tried the .exe and it also doesn't start. I have created the desktop folders, so I'm obviously missing something?

Per the dump log above, which .NET version did you use to compile?

wwaag wrote on 3/18/2020, 9:40 PM

All I can say is--something's definitely wrong, but you already knew that. I went to another machine, downloaded the package, and both the Magix and Sony versions worked in V14 and V13 versions, respectively. Additionally, I've had a couple of other users try it without such issues.

Just looked at the error message once again which vaguely rings a bell. My suspicion is that Windows has blocked your files. For both the dll and exe, right click and select properties and make sure that it is not blocked. If blocked, it will look something like this.

If blocked, click on Unblock and then Apply.

john_dennis wrote on 3/18/2020, 10:23 PM

@Musicvid

Fear not. It can happen.

john_dennis wrote on 3/18/2020, 11:07 PM

@Musicvid @wwaag

I tried it in Vegas Pro 14 and got the following error.

Incidentally, I put RenderQuality.dll and RenderQuality.png in this path...

Vegas Pro 15 and 17 work as designed. Even when pointing to the .dll in the Vegas 14 folder.

Musicvid wrote on 3/19/2020, 12:43 AM

john_dennis wrote on 3/18/2020, 10:07 PM

@Musicvid @wwaag

I tried it in Vegas Pro 14 and go the following error.

I am running VP14.

fifonik wrote on 3/19/2020, 2:04 AM

Downloaded, unzipped, run and got an error.

In error the program shown path that does not exist, but that was used long time ago with previous version (that was uninstalled).

So it was stored in registry and current program just tried to use it without checking that the directory exist.

I deleted entries and everything looks good for now.

JN- wrote on 3/19/2020, 8:15 AM

@wwaag Thanks for the version 2.

I attempted to run on my laptop but got an eror message "Object reference not set to an instance of an object".

Anyway I attempted to use Ver 2 the older way by creating the .bmp clips and simply running RenderQualityMetrics.exe.

I tested only 10 frames, leaving out SSIM for the moment.

When I got what looked like odd results I ran the ver 1 Render quality tool and compared, the results aren't the same?

Am I doing something wrong here, can ver 2 be run as standalone as I have tried?

Also, can tiff's be used instead of .bmp's?

I ran Ver 2 with 51 tiff's = ~2 seconds and got similarily odd results ...

In this 2nd. test I used the same 51 stills in both input and output folders.

MSE ……….... PSNR … 51 stills used ...

15,927.709 …….. 6.109

The results from ver 1 was as expected ...

MSE …….. PSNR

0 ....... 99

Just got the results in from a full 27s test, 675 .bmp frames, all tested, using ver 1 and ver 2 as standalones.

This was from a MC CPU render that previously gave ...

Ver 1 ... MSE 8.719 … PSNR 40.099

Ver 2 … MSE 17,726.756 … PSNR 5.816 … SSIM 0.0418

wwaag wrote on 3/19/2020, 11:37 AM

What a mess. I have taken down the new version until it's sorted out. The problems stem from use of the same registry locations as the original version.

@JN-

V1 and V2 are not compatible. In V1 differences are computed within the app itself. In V2, differences are computed inside of Vegas and only a single image reflecting those differences (an almost black image) is exported. That would explain those humongous differences

At the moment, it does not do Tif's. I can easily add.

wwaag wrote on 3/19/2020, 11:55 AM

@JN-

You can already use tiff's, png's or jpg's in an addition to bmp's. Just select it in the FrameServer dialog as shown. I tried all 4 and they work. Bmp's, png's and tiff's give the exact same result, while jpg's are a little different. The main difference is that BMPs are much faster when rendered from Vegas.

JN- wrote on 3/19/2020, 1:43 PM

@wwaag Thanks for clearing some of that up.

So v2 cannot be used standalone. Tiff can be used in ver 2. I never installed your HO, frameserver etc, waiting for full product release.

When you release v2 update I’ll try again, hopefully the error dialog won’t appear “object reference ...”

Do I need your main program installed to avoid this error message? If so, of course I’ll install as this is a great addition to the render quality tool.

My reason for preferred use of TIFF over BMP is that the latter is painfully slow on my PC, but not my laptop. Before you gave the go ahead to use TIFF in ver 1, I found a workaround by using a ram drive for PC testing. This is just a weird peculiarity of my PC.

wwaag wrote on 3/19/2020, 3:49 PM

I've just uploaded a new version which hopefully fixes the install problems. The direct link can be found here https://tools4vegas.com/render-quality/.

Note that you MUST install the DebugMode FrameServer(DMFS). Although recommended since it installs DMFS for all Vegas versions (V12-V17) on your system, there is no firm requirement to install Happy Otter Scripts. You can download DMFS directly at https://www.debugmode.com/frameserver/.

Musicvid wrote on 3/19/2020, 4:48 PM

[EDIT] It looks like its running after installing the new .dll and restarting the computer.

More to come.

JN- wrote on 3/19/2020, 5:48 PM

@wwaag I installed frameserver aok. Ran a new test using ver 2, I noticed that you cannot rename this .exe, or it won’t be found. So success doing the test, but my results are still somewhat different to what I previously had. I cannot double check on the same machine running ver 1, error reported from it and I will test ver 1 results on my other machine. I expect that when I do it will be what I got several months ago.

Anyway, using the new ver 2, I got ... MSE 1.7 (8.719) ... PSNR 46.785 (40.099) ... SSIM 0.9226 (N/A)

The values in brackets are from ver 1 previously, so the MSE is much different.

Not sure what size “block” to use, I used 8x8 for the above test, used BMP. I'm running another test, its taking a while, using tiff, 675 frames, 1x1 block.

Musicvid wrote on 3/19/2020, 6:51 PM

It's a keeper Wayne, thanks!

wwaag wrote on 3/19/2020, 7:00 PM

@JN-

If you want truly accurate results you should use 1x1 which basically looks at every pixel. This is what version 1 does. The estimated values will be larger for the larger block sizes. Whatever block size you choose, Vegas will reduce the frame size by that factor. E.g. if you choose 8x8 for an HD project, the frame size of the rendered BMPs will be 1920/8 x 1080/8 which equals 240 x 135 which isn't a lot of pixels. This is the reason that estimates of PSNR in FFmpeg are always much higher in that it reportedly uses an 8 x 8 block size, which accounts for why it is so fast in comparing two rendered files. Once the frameserver window appears, you can tell exactly the new frame dimensions by dragging it away from the preview window. It's a lot faster, but the results are an over-estimate of the true value. The user has to decide on which value is appropriate for his needs. In my view, it's not the "true" value that matters but the relative value across different renders. So long as the user sticks with the same value, it should be OK. The same is true in choosing value for "every nth pixel".

JN- wrote on 3/19/2020, 7:25 PM

@wwaag Thanks for that clarification. Its still running on the 27s clip 1x1 block, now I know why its still only at frame 530 of frame 675🤣. I’ll use 1x1 for all future tests. I assume that the “difference” setting/test is done automatically by the script, or do I need to do it?

”The same is true in choosing value for "every nth pixel" Cannot check that now, still running, is that a user selectable item and if so what gives the best, most accurate result? Ok, I remember now, default is 1, I will leave at 1 for any future testing, thanks.

fifonik wrote on 3/19/2020, 7:42 PM

This is the reason that estimates of PSNR in FFmpeg are always much higher in that it reportedly uses an 8 x 8 block size, which accounts for why it is so fast in comparing two rendered files.

Could you clarify, where have you got the information please?

P.S. I wish Measuring Render Quality would be standalone and could accept avs files.

JN- wrote on 3/19/2020, 8:03 PM

@wwaag Results just came in, the PSNR and MSE values are identical to what I had previously got using ver. 1 of HOQM, brilliant, excellent.

The SSIM value I got is 0.6752. The ffmpeg value (ffmpeg SSIM "ALL") for the same clip previously tested was 0.984235.

The clip I used is number 13 in the screenshot below. I'll update this chart with the HOQM SSIM values in the future, will take some time.

Is this big difference in ffmpeg SSIM vs HOQM SSIM what you would expect?

I've added a table using FHD instead of UHD clips below, it allows comparison of the older Sony AVC codec.

UPDATE: I've moved the tables with the results to the second page of this thread.

wwaag wrote on 3/19/2020, 9:00 PM

@JN-

Regarding "every nth pixel"--this value is not nearly as important. I'd suggest trying different values. I've found that the results are pretty much the same whether using every 1, 5th or 10th. Here, statistical sampling is your friend. For the demo, I used every 5th frame.

@fifonik

"Could you clarify, where have you got the information please?" I don't recall exactly. I did a lot of searches involving psnr and ffmpeg. I believe it was from one of the many threads on either StackOverflow or SuperUser.

I actually wrote another standalone version in which you could simply input two rendered files using FFmpeg, but I found the results were just too inconsistent to be usable--e.g. a file rendered to Cineform or even Uncompressed showed huge differences with the original file as the "standard".

I've also "played around" with different Avisynth scripts using AviDub. I have one which writes the average Luma for Y, U, and V to a log file which in turn could be easily be processed for summary measures. If you input the difference from the track compositing approach, you actually get a good estimate of render quality for each frame, although I have no idea exactly "what" the metrics represent. Once I have some time, I want to pursue this further since it does represent a very "fast" approach.

I should also add that in the next build of HOS, I have included measures of psnr and ssim when available. Psnr is available for x264, x265, and vp9 renders while both psnr and ssim are available for Nvenc and VCE renders.

wwaag wrote on 3/19/2020, 9:24 PM

@JN-

That's good news that they are the same. That's what I found as well so I at least know that the use of the difference from Vegas is the same as computing the differences externally.

Regarding the difference between FFmpeg SSIM and the measures from RQM, I have no idea. My suspicion is that the FFmpeg estimates are inflated due to their block size sampling. Given how quickly FFmpeg computes these measures, it doesn't seem reasonable that they are looking at every pixel. I should add that in RQM, the SSIM calculations always looks at all pixels. There is no sampling.

JN- wrote on 3/19/2020, 9:53 PM

@wwaag 👍 Sounds about right, thanks for the work u did on this, a very useful addition indeed.

fifonik wrote on 3/19/2020, 10:05 PM

I actually wrote another standalone version in which you could simply input two rendered files using FFmpeg, but I found the results were just too inconsistent to be usable--e.g. a file rendered to Cineform or even Uncompressed showed huge differences with the original file as the "standard".

I was there when did my comparison using VQMT. Some encoders shifts video stream for a few frames while rendering (@john_dennis found this in another thread as well) and this can only be corrected manually.

wwaag wrote on 3/19/2020, 10:24 PM

So that's the problem. Interesting. Thanks for the info.

"and this can only be corrected manually."

Actually, you could probably do this in code. In SmartVideoTrim, if an exact frame match cannot be found between the original and the "copy", it enters what I term "Deep Analysis" in which differences are calculated and the frame selected which produces the minimum difference is the one selected as the "match". This technique could be applied to find the frame offset between between two files. Having said that, I doubt it would be worth the effort.

That's the beauty of doing it inside of Vegas. The user can easily make sure that the frames of the rendered and original are "aligned" before starting the analysis.

john_dennis wrote on 3/19/2020, 11:27 PM

"Some encoders shifts video stream for a few frames while rendering (@john_dennis found this in another thread as well)"

For reference:

https://www.vegascreativesoftware.info/us/forum/what-is-the-best-format-to-choose-for-youtube-when-do-video-rendering--119333/#ca744728