Tool for Measuring Render Quality - Version 2

Comments

john_dennis wrote on 4/26/2020, 4:49 PM

I decided to make a Howie Duit to describe the process for installing the Render Quality Metrics tool on Vegas Pro 17.

Since I won't be taking my wife to dinner, I made a Howie Duit on installing Happy Otter Scripts, also.

lenard-p wrote on 4/26/2020, 6:25 PM

I downloaded av1,avc and vp9 versions of a youtube video, Is anyone able to check what is best quality?

https://drive.google.com/open?id=1MCoLZQSp2NVYPvmBpSK6OKpQGZ-Fg38j

Musicvid wrote on 4/26/2020, 8:10 PM

@lenard-p

You cannot compare three compressed renders against each other and determine which is "best" because there is no uncompressed reference, it's just a butterfly chase without it.

JN- wrote on 4/26/2020, 8:23 PM

@lenard-p As a preliminary guess, assuming that the largest file has the highest data rate, best quality i.e. the .mp4. Mediainfo cannot open the .avi, but given it's small size, I'm guessing its not the highest data rate of the 3.

So I did a very quick ffmpeg test, (can only open .mp4 in VP, so no HORQ test possible) based on the my guess that the .mp4 being the best quality then the order is 1st. .mp4, 2nd. best avi and in 3rd place the .mkv. file.

ffmpeg ... higher is better ...

AVI SSIM ALL 0.965656 (14.641506) … PSNR 29.638459

MKV SSIM ALL 0.963861 (14.420256) … PSNR 29.402949

The above all falls apart if my starting assumption is incorrect, that the .mp4 file has the best quality, chasing butterflies, or mice in a field, take your pick :D.

Last changed by JN- on 4/27/2020, 4:42 AM, changed a total of 1 times.

---------------------------------------------

Benchmarking thread

Codec Render Quality tables

---------------------------------------------

PC ... Corsair case, own build ...

CPU .. i9 9900K, iGpu UHD 630

Memory .. 32GB DDR4

Graphics card .. MSI RTX 2080 ti

Graphics driver .. latest studio

PSU .. Corsair 850i

Mboard .. Asus Z390 Code

 

Laptop ... (Acer Predator G9-793-77AC)

CPU .. i7-6700HQ Skylake-H

Memory ..32 GB DDR4, was previously 16 GB

Graphics card .. Nvidia GTX 1070

Graphics driver .. latest studio

wwaag wrote on 4/26/2020, 9:11 PM

@Musicvid is absolutely right. You need some type of uncompressed reference--i.e. the original or if it's a render from Vegas at least a lossless render for comparison. Even under such circumstances, the comparison may or may not be correct given the way YT mangles video. For example, if the two files are a single frame off (i.e. not synced) the resulting errors will be very large. If you upload a constant frame rate video and YT converts it to VFR during encoding, there is another source of error. Another source of error concerns different containers. Since the presentation time stamps ( referred to as PTS) can be different across containers, again a seemingly simple comparison can lead to erroneous results. Having listed just a few of the "gotchas", you can make comparisons, but you need to be very careful.

Musicvid wrote on 4/26/2020, 10:02 PM

Also, comparisons of progressive renders to interlaced source are worthless for the same reasons.

GJeffrey wrote on 4/27/2020, 4:17 AM

Below my result using Wayne's RQMv2 tool using different codecs available in Vegas and HOS.

The original file (108025p AVC from GH4) can be downloaded here

RQM using1x1 block, select every frame. I use only the first 100 frame of the above file for RQM purpose.

Playback rate with Vegas 17 latest version, Hardware decoding on, using a very old PC (i7 3930K, GTX1070ti, file on HDD)

AVC and HEVC encoding (Magix, ffmpeg & Nvenc) using VBR rate max 26Mbps, target 25Mbps to match SonyAVC result.

Nvenc lossless looks a very interesting alternative to MagicYUV or Ut codec. Magix intermediate is very good as well but doesn't play well on my computer.

@wwaag Could it be possible to implement nvenc lossless encoding in HOS RE & R+?

 

JN- wrote on 4/27/2020, 4:47 AM

@GJeffrey Hi GJeffrey, nice. Be aware that the magix lossless is broken, corruption issues etc in my testing using UHD. Some findings I previously had here at this link.

Your results confirm mine for Sony avc pecking order.

Last changed by JN- on 4/27/2020, 4:57 AM, changed a total of 1 times.

---------------------------------------------

Benchmarking thread

Codec Render Quality tables

---------------------------------------------

PC ... Corsair case, own build ...

CPU .. i9 9900K, iGpu UHD 630

Memory .. 32GB DDR4

Graphics card .. MSI RTX 2080 ti

Graphics driver .. latest studio

PSU .. Corsair 850i

Mboard .. Asus Z390 Code

 

Laptop ... (Acer Predator G9-793-77AC)

CPU .. i7-6700HQ Skylake-H

Memory ..32 GB DDR4, was previously 16 GB

Graphics card .. Nvidia GTX 1070

Graphics driver .. latest studio

GJeffrey wrote on 4/27/2020, 5:16 AM

@JN-

Nvenc lossless is using HOS with a custom command actually, not the internal magix nvenc encoder

JN- wrote on 4/27/2020, 5:28 AM

@GJeffrey Ok, so thats a workaround I guess. I was curious obviously, and would have liked to test it with the other lossless and intermediates, but couldn’t. So you’ve done that now. I haven’t used HOS, waiting for GODOT.🤣

This was the only mention from Magix @VEGASHeman , I guess their still looking into it, maybe its a very deep well.

“@JN_: We will look into the 0-byte UHD NVENC lossless renders you are seeing. I believe BruceUSA is referring to Magix Intermediate 422 format, which is different from this NVIDIA specific lossless renders. I will also contact you for more details about the artifact you see in the H264 render on your 1070.”

For the record, I never heard back from anyone in Magix, but hey, it’s early days yet, that was only 10 months ago.

 

Last changed by JN- on 4/27/2020, 7:51 AM, changed a total of 2 times.

---------------------------------------------

Benchmarking thread

Codec Render Quality tables

---------------------------------------------

PC ... Corsair case, own build ...

CPU .. i9 9900K, iGpu UHD 630

Memory .. 32GB DDR4

Graphics card .. MSI RTX 2080 ti

Graphics driver .. latest studio

PSU .. Corsair 850i

Mboard .. Asus Z390 Code

 

Laptop ... (Acer Predator G9-793-77AC)

CPU .. i7-6700HQ Skylake-H

Memory ..32 GB DDR4, was previously 16 GB

Graphics card .. Nvidia GTX 1070

Graphics driver .. latest studio

Musicvid wrote on 4/27/2020, 9:11 AM

As mentioned, 1x1 may be too many trees and not enough forest. See if 8x8 raises the scores closer to the keystone reference, which is the MSU quality tool.

Musicvid wrote on 6/25/2020, 6:50 PM

[Edited]

@wwaag

wwaag wrote on 6/25/2020, 9:03 PM

@Musicvid

Just ran about 10 RQ's in V14 and it worked without issue. The error message you posted--when does it occur? Basically, that message means that the program is expecting something that is not there.

Musicvid wrote on 6/25/2020, 9:21 PM

Yes, when I press

If there are files in the folder, it offers to delete them. Then the same message appears.

I'm pretty sure it's not finding Frameserver, although FS will run and create a signpost file from within Vegas. HOS is not installed at this time.

wwaag wrote on 6/25/2020, 9:37 PM

Try deleting the existing bmp files manually and then run. It probably won't make a difference, but that would rule out that part of the code.

wwaag wrote on 6/25/2020, 9:41 PM

Also give me a screen grab of the frameserver registry entries. Here is mine.

Musicvid wrote on 6/25/2020, 9:47 PM

here's mine

wwaag wrote on 6/25/2020, 9:49 PM

No, I need the FrameServer registry entries.

Musicvid wrote on 6/25/2020, 9:54 PM

wwaag wrote on 6/25/2020, 10:08 PM

I don't see anything that would cause the problem. Try this. Do a RenderAs using the frameserver and actually render out an image sequence somewhere. Then try RQ again.

Musicvid wrote on 6/25/2020, 10:14 PM

No luck. I'm going to turn in. Lots of time available tomorrow.

wwaag wrote on 6/26/2020, 11:25 AM

On my second system, I uninstalled HOS which removes the FrameServer, installed FrameServer 3.0 using the normal setup file, and then installed the RQ package. It is important to first "run" RQMv2.exe to set up your folder locations since it writes that info to the registry. I then addded the script to the V14 script folder, and unfortunately for you, it worked just fine.

My suggestion at this point would be to uninstall DMFS and RQ including the registry and "start over".

If it still isn't working, I'll see about creating a special exe file with some additional diagnostics to determine exactly where it is failing on your system.

Musicvid wrote on 6/26/2020, 12:44 PM

I did uninstall both programs and delete the registry folders yesterday. But there's no harm in trying again. Stay tuned.

Musicvid wrote on 6/26/2020, 7:13 PM

I deep-uninstalled Frameserver and removed the registry folder. Same with RQM.

Same results upon reinstalling both apps.

Wayne, I'm convinced it is with the Frameserver. I see the signpost file when I start it manually, but there is no video content, just black. The Frameserver dialog never appears. RQM can't find it. the FS plugin is in FileIO Plug-Ins. I downloaded 3.0 64 bit. Is there an older version I should be using instead?