Tool for Measuring Render Quality - Version 2

wwaag wrote on 3/18/2020, 3:35 PM

Over a year ago, I started a thread https://www.vegascreativesoftware.info/us/forum/tool-for-measuring-render-quality--114398/?page=1 which described a tool, Render Quality Metrics (RQM) for measuring the quality of renders from Vegas. Admittedly, RQM was pretty "clunky" in that it required the user to render an image sequence of BMPs for both the Original and Rendered files. A new version is available in the Happy Otter Free Tools Library which greatly simplifies and speeds up its use. In addition to Mean Squared Error (MSE) and Power Signal to Noise Ratio (PSNR), RQM now has an option to compute the Structural Similarity Index Metric (SSIM) although it is very slow. RQM now has two apps, a script that renders the image sequence from Vegas and a standalone app that does the computation.

Rather than render two sets of BMPs, RQM makes use of the track compositing mode in Vegas. The rendered file is placed on the top track and its compositing mode is set to "Difference". Another difference is that the BMPs are now directly rendered using the DebugMode FrameServer (DMFS) which speeds up rendering. I've done testing to make sure that BMPs from DMFS are identical to those produced by Vegas. I've also done testing to make sure that the single BMP based on the composite difference in Vegas is the same as the difference from separately rendered BMPs.

There are also options to "speed up" the process by selecting the pixel block size during creation of the BMPs and also an option to select every nth frame during processing. Here are screen grabs of both the script and standalone app.

A demo of its usage can be found at https://vimeo.com/398614859

The new version is titled Vegas Render Quality and can be downloaded at https://tools4vegas.com/library/ The zip file contains both the script and the standalone app.

Comments and suggestions for improvement are welcome.

Back to post

Comments

john_dennis wrote on 4/26/2020, 4:49 PM

I decided to make a Howie Duit to describe the process for installing the Render Quality Metrics tool on Vegas Pro 17.

Since I won't be taking my wife to dinner, I made a Howie Duit on installing Happy Otter Scripts, also.

Former user wrote on 4/26/2020, 6:25 PM

I downloaded av1,avc and vp9 versions of a youtube video, Is anyone able to check what is best quality?

https://drive.google.com/open?id=1MCoLZQSp2NVYPvmBpSK6OKpQGZ-Fg38j

Musicvid wrote on 4/26/2020, 8:10 PM

@Former user

You cannot compare three compressed renders against each other and determine which is "best" because there is no uncompressed reference, it's just a butterfly chase without it.

JN- wrote on 4/26/2020, 8:23 PM

@Former user As a preliminary guess, assuming that the largest file has the highest data rate, best quality i.e. the .mp4. Mediainfo cannot open the .avi, but given it's small size, I'm guessing its not the highest data rate of the 3.

So I did a very quick ffmpeg test, (can only open .mp4 in VP, so no HORQ test possible) based on the my guess that the .mp4 being the best quality then the order is 1st. .mp4, 2nd. best avi and in 3rd place the .mkv. file.

ffmpeg ... higher is better ...

AVI SSIM ALL 0.965656 (14.641506) … PSNR 29.638459

MKV SSIM ALL 0.963861 (14.420256) … PSNR 29.402949

The above all falls apart if my starting assumption is incorrect, that the .mp4 file has the best quality, chasing butterflies, or mice in a field, take your pick :D.

wwaag wrote on 4/26/2020, 9:11 PM

@Musicvid is absolutely right. You need some type of uncompressed reference--i.e. the original or if it's a render from Vegas at least a lossless render for comparison. Even under such circumstances, the comparison may or may not be correct given the way YT mangles video. For example, if the two files are a single frame off (i.e. not synced) the resulting errors will be very large. If you upload a constant frame rate video and YT converts it to VFR during encoding, there is another source of error. Another source of error concerns different containers. Since the presentation time stamps ( referred to as PTS) can be different across containers, again a seemingly simple comparison can lead to erroneous results. Having listed just a few of the "gotchas", you can make comparisons, but you need to be very careful.

Musicvid wrote on 4/26/2020, 10:02 PM

Also, comparisons of progressive renders to interlaced source are worthless for the same reasons.

GJeffrey wrote on 4/27/2020, 4:17 AM

Below my result using Wayne's RQMv2 tool using different codecs available in Vegas and HOS.

The original file (108025p AVC from GH4) can be downloaded here

RQM using1x1 block, select every frame. I use only the first 100 frame of the above file for RQM purpose.

Playback rate with Vegas 17 latest version, Hardware decoding on, using a very old PC (i7 3930K, GTX1070ti, file on HDD)

AVC and HEVC encoding (Magix, ffmpeg & Nvenc) using VBR rate max 26Mbps, target 25Mbps to match SonyAVC result.

Nvenc lossless looks a very interesting alternative to MagicYUV or Ut codec. Magix intermediate is very good as well but doesn't play well on my computer.

@wwaag Could it be possible to implement nvenc lossless encoding in HOS RE & R+?

JN- wrote on 4/27/2020, 4:47 AM

@GJeffrey Hi GJeffrey, nice. Be aware that the magix lossless is broken, corruption issues etc in my testing using UHD. Some findings I previously had here at this link.

Your results confirm mine for Sony avc pecking order.

GJeffrey wrote on 4/27/2020, 5:16 AM

@JN-

Nvenc lossless is using HOS with a custom command actually, not the internal magix nvenc encoder

JN- wrote on 4/27/2020, 5:28 AM

@GJeffrey Ok, so thats a workaround I guess. I was curious obviously, and would have liked to test it with the other lossless and intermediates, but couldn’t. So you’ve done that now. I haven’t used HOS, waiting for GODOT.🤣

This was the only mention from Magix @VEGASHeman , I guess their still looking into it, maybe its a very deep well.

“@JN_: We will look into the 0-byte UHD NVENC lossless renders you are seeing. I believe BruceUSA is referring to Magix Intermediate 422 format, which is different from this NVIDIA specific lossless renders. I will also contact you for more details about the artifact you see in the H264 render on your 1070.”

For the record, I never heard back from anyone in Magix, but hey, it’s early days yet, that was only 10 months ago.

Musicvid wrote on 4/27/2020, 9:11 AM

As mentioned, 1x1 may be too many trees and not enough forest. See if 8x8 raises the scores closer to the keystone reference, which is the MSU quality tool.

Musicvid wrote on 6/25/2020, 6:50 PM

[Edited]

@wwaag

wwaag wrote on 6/25/2020, 9:03 PM

@Musicvid

Just ran about 10 RQ's in V14 and it worked without issue. The error message you posted--when does it occur? Basically, that message means that the program is expecting something that is not there.

Musicvid wrote on 6/25/2020, 9:21 PM

Yes, when I press

If there are files in the folder, it offers to delete them. Then the same message appears.

I'm pretty sure it's not finding Frameserver, although FS will run and create a signpost file from within Vegas. HOS is not installed at this time.

wwaag wrote on 6/25/2020, 9:37 PM

Try deleting the existing bmp files manually and then run. It probably won't make a difference, but that would rule out that part of the code.

wwaag wrote on 6/25/2020, 9:41 PM

Also give me a screen grab of the frameserver registry entries. Here is mine.

Musicvid wrote on 6/25/2020, 9:47 PM

here's mine

wwaag wrote on 6/25/2020, 9:49 PM

No, I need the FrameServer registry entries.

Musicvid wrote on 6/25/2020, 9:54 PM

wwaag wrote on 6/25/2020, 10:08 PM

I don't see anything that would cause the problem. Try this. Do a RenderAs using the frameserver and actually render out an image sequence somewhere. Then try RQ again.

Musicvid wrote on 6/25/2020, 10:14 PM

No luck. I'm going to turn in. Lots of time available tomorrow.

wwaag wrote on 6/26/2020, 11:25 AM

On my second system, I uninstalled HOS which removes the FrameServer, installed FrameServer 3.0 using the normal setup file, and then installed the RQ package. It is important to first "run" RQMv2.exe to set up your folder locations since it writes that info to the registry. I then addded the script to the V14 script folder, and unfortunately for you, it worked just fine.

My suggestion at this point would be to uninstall DMFS and RQ including the registry and "start over".

If it still isn't working, I'll see about creating a special exe file with some additional diagnostics to determine exactly where it is failing on your system.

Musicvid wrote on 6/26/2020, 12:44 PM

I did uninstall both programs and delete the registry folders yesterday. But there's no harm in trying again. Stay tuned.

Musicvid wrote on 6/26/2020, 7:13 PM

I deep-uninstalled Frameserver and removed the registry folder. Same with RQM.

Same results upon reinstalling both apps.

Wayne, I'm convinced it is with the Frameserver. I see the signpost file when I start it manually, but there is no video content, just black. The Frameserver dialog never appears. RQM can't find it. the FS plugin is in FileIO Plug-Ins. I downloaded 3.0 64 bit. Is there an older version I should be using instead?