ffmpeg ssim results are all over the map, like chasing butterflies.

For that reason I don't use it.

Thanks to John Dennis and his uncanny ability to sniff out Otter bugs, I've uploaded a new version that can be downloaded at https://tools4vegas.com/render-quality/ In the event the test folder no longer exists, the user has the option to select a different a different folder or simply create a new folder with the existing name. There is also a check to make sure that the test folder is not empty. Please advise of any additional problems you might encounter or suggestions for improvement.

HOS was fundamentally created as a Bridge. A Bridge Between that which was opaque to someone, like me, and thus made transparent and workable. RE and KP being two show stoppers.

Presently, 93.7245% of what is being discussed, in this high level programming-rich thread, scares the pants off me for making being aware of how little I do know about this stuff. Bottom line, I’m in total awe at those who know and understand this process.

KUDOS to Wayne and Team-HOS! Just ‘cos I don’t understand doesn’t mean that I’m ungrateful to all the efforts being expended on this “Quality” project. 😀

Ultimately, can we have a Grazie-Proof version?

@wwaag I'm getting this error message on my other, not main machine, small square dialog error box ...

“value cannot be nul

parameter name: value”

It appears immediately after all of the .bmp's are written, but before I can use, click on the HOQM ver 2 tool. Any ideas? It’s an up to date win 10 1909 PC.

@wwaag Put a hold on that, working now without the error message. I checked updates again and got a .NET something, perhaps that fixed it.

I did a simple test.

I rendered to ProRes 422 and ran Wayne's SSIM at 1x1 and 8x8.

The 1x1 came in about 5 points lower than the 8x8. The 8x8 result was identical to the venerable MSU QMT (± 1%).

This suggests to me that perhaps the 1x1 granularity is too fine for practical use, if two other tools "may" be using ≈8x8 blocks, speed notwithstanding. Maybe set 8x8 as the default?

Anyway, at 8x8 and every other frame processing, the pace is quite easy to take. Thanks again, Wayne!

@wwaag

“Regarding "every nth pixel"--this value is not nearly as important. I'd suggest trying different values. I've found that the results are pretty much the same whether using every 1, 5th or 10th. Here, statistical sampling is your friend. For the demo, I used every 5th frame.”

Is the above referring to where I selected every single frame, thanks. Or is there a user selectable setting for the “every nth pixel", if so I may have missed it, thanks.

Use the numeric up/down box to select how many frames to include. E.g. if 5 is entered as shown, every 5th frame will be analyzed.

@wwaag Thanks. Great tool to have.

@wwaag I'm getting this message again "Object reference not set to an instance of an object". when I try to run the Render quality script. Haven't used the RQM tool for a while, Debug frameserver is still installed and the "Paths" in RenderQualityMetricsV2.exe when I run it are valid, still the same. I checked it on my laptop and PC, same issue. Any ideas? could some intervening Win 10 updates have changed something? Thanks.

@JN-

Exactly, when does the message appear? During the initial FrameServer operation from the timeline? Or when you open the RQM app?

Just ran RQ a bunch of times without issue. That error message indicates that the program expects somethings that isn't there--most likely, a specific folder. Make sure that the folder containing the BMPs exist and also that the log folder. As I recall, I put in a check to make sure the BMP folder exists, but I'm not sure about the log folder. If you click on the icon with the folder and glass, it should open the log folder.

If everything is OK, I'd suggest "starting over" by deleting the following registry key using Regedit.

Computer\HKEY_CURRENT_USER\Software\HappyOtterTools\RenderQualityMetrics_V2

Then run RQM to set up your folders and try again.

@wwaag Thanks for getting back. Ok, what I do is load up my source file in VP17,

1) render out the bmp's from it to this … D:\HOQM\Render Quality\Original\

This is a screenshot of the paths …

2) Then I put the file to test on the top track, above my source file, go to tools/scripting and select your script RenderQuality.

I leave it at defaults ...

and click start, then I get the error message.

Ok, I think that step 1) above is not needed.

I'll have a go at removing the registry entries tomorrow. This is whats entered currently under version 2 ...

Anyway, deleted the Reg entries, set up again, as above, still no joy.

I'm confused by "exactly" what you are doing.

In (1) "render out the bmp's from it to this". Are you trying to render the bmp's like in the first version? If so, that's not necessary. Have you verified that those two folders actually exist?

(2) So the error occurs once you click "start" inside the Vegas script?

(3) Does it render the BMP's?

(4) Have you tried to start the frameserver from the normal RenderAs just to make sure that it is installed correctly?

By far, the best diagnostic tool is a screen recording which is easy to do in V17.

@wwaag Its looking good, I'm currently past the point where the error dialog appears, waiting for the results.

What did I do differently? I used VP16, as I had for all of my previous testing. Today I was using only VP17, and so the error.

"In (1) "render out the bmp's from it to this". Are you trying to render the bmp's like in the first version? If so, that's not necessary."

In my previous post I did see the error of my ways, and already mentioned that that was a bit redundant.

"By far, the best diagnostic tool is a screen recording which is easy to do in V17."

So that's changed for the better? When I last tested it, on release, it looked like something that was a serious afterthought. In my worst attempts at programming, in the distant past, when I had some brain cells, I'd have seriously considered a name change if it was mine.

"What did I do differently? I used VP16, as I had for all of my previous testing. Today I was using only VP17, and so the error."

Aha! That's the problem. You probably don't have the required FrameServer file copied into the V17 folder. Here's what to do. Go to "C:\Program Files\VEGAS\VEGAS Pro 16.0" and copy the following file "Frameserver.x64.fio2007-config". Then paste it into "C:\Program Files\VEGAS\VEGAS Pro 17.0" . Then try it.

@wwaag Ok, that completed, one down, seven to go. I'm doing a quite smaller number of FHD test clips, previously I did only UHD. The reason is there is often some debate about how good or otherwise the old Sony render codec is. It doesn't do higher than FHD, ~20 Mbps, so I am testing it with Magix CPU, Nvenc-gtx+rtx, MC-Legacy, ffmpeg, Xavc-s long, Xavc-I.

Thanks for all of the suggestions WWAAG, the error issue appears to be tied into VP17, on my end anyway.

This table below uses FHD test clips. It's divided into several sections. I've added in a few higher data rates above the black line, plus a few Lossless and intermediate samples in the top section of the table.  It's main function initially was to determine how the older Sony avc codec compared to the more recent codecs, it's in the bottom section, item no. 27.

#08.

#09. The table below use UHD test clips, general files comparison.

#10. The table below uses UHD test clips, Intermediate multi generation comparison.

Link to all screenshots … here

FWIW While rendering out to the QSV codec for the above FHD tests I came across a corruption issue in the output clip, using VP16 and VP17. When I tested on a UHD clip, I had similar problem, and solution, see below.

I was using my main, "Corsair" PC, (see profile). It allows up to a QSV quality setting of max 7. I got around the problem by using an older PC, CPU 4790K, (doesn't use, it's not DCH capable) that also allowed a render quality max of 7, no issue there.

I checked this out again in the Corsair PC, I updated the Intel/Nvidia drivers to the very latest today, (Intel is DCH type) no change.

What did fix the issue was setting the quality setting to 6 or less, then no corruption. I know that when doing previous QSV tests, say a year ago, there was no problem, for quality testing I would have always used the highest setting. Probably the case that if I reverted back to the "standard", non-DCH driver on my main PC, then I wouldn't have any issue, couldn't be bothered.

Since I don't use QSV for rendering anyway, (CPU or Nvenc, I find better quality) It's no big deal for me, just putting it out there in case anyone else were to come across it.

@wwaag I know that I already raised this query, but the discrepancy at the lower end of the SSIM values between ffmpeg and your HORQ tool is really large in the FHD, top most table above. The difference appears to accelerate as the quality goes down, for example the Sony Avc codec. The UHD tables differences are not that noticeable, because maybe the data rates are higher.

Are you still ok with the results.

On the issue of my not being able to use the HORQ tool in VP17, its ok for me in VP16, can I take it that when you said you had no problem that you were using VP17?

Thanks.

@JN-

Regarding V17, did you add the file to the V17 folder as suggested in a previous in my last response? Most likely that's the problem although there should be no difference results between the 2 versions.

Regarding differences with FFmpeg, most likely it is the difference that FFmpeg computes these values. Unlike the SSIM calculations in RQ, apparently they do not do comparisons on a per pixel basis. From what I've read, they use an 8x8 block of pixels to make estimates which would explain why it is fast in comparison with RQ. I have another version with just FFmpeg doing the calculations and I finally abandoned it due to the wild discrepancies--e.g. very large errors with codecs that are known to be very high quality such as Cineform.

@wwaag Thanks for getting back on this. Have you tested the tool on VP17?

Frameserver is installed, the Tool runs aok in VP 16 on my PC and Laptop, doesn't run as previously reported on my PC or Laptop on VP17.

This is where I copied the 2 files to for VP17
C:\Program Files\VEGAS\VEGAS Pro 17.0\Script Menu\RenderQuality.dll
C:\Program Files\VEGAS\VEGAS Pro 17.0\Script Menu\RenderQuality.png

------------------------------------------------------------------------------------------------------

RE: SSIM. It also struck me that as well as the lower end encoders, Sony avc etc having a very low SSIM with your tool compared to ffmpeg, that at the higher end, MagicYUV, mathematically lossless, only gets to 0.98 ... ffmpeg rates it at 0.99 ...

So I tested the source file against the source file in HORQ-2 and it does appear to be ok ...

MSE = 0, PSNR = 99 and importantly SSIM = 1.

@JN-

Here's what I wrote.

"Aha! That's the problem. You probably don't have the required FrameServer file copied into the V17 folder. Here's what to do. Go to "C:\Program Files\VEGAS\VEGAS Pro 16.0" and copy the following file "Frameserver.x64.fio2007-config". Then paste it into "C:\Program Files\VEGAS\VEGAS Pro 17.0" . Then try it."

Have you done this?