You may wish to objectify your visual impressions with carefully controlled SSIM and PSNR tests. There are a few ways to do this.

@Musicvid +1

There's a tool, Vegas Render Quality, in the HOS Free Tools Library that computes mse, psnr, and ssim. Here's a link to a demo.

https://vimeo.com/398614859

And a link to the download page.

https://tools4vegas.com/render-quality/

There are also tools available using FFmpeg which IMHO are not as accurate due to their usage of large block sizes in their computation. Also, if you use the RenderPlus tool in HOS, most codecs allow for calculation of both PSNR and SSIM as part of their output.

@andyrpsmith Here's a link to an ffmpeg-based tool, ffmetrics. I'd be curious to hear how the rankings of the alternative mechanical methods compare to that of your eyes. Particularly, the mse rating computed by @wwaag's tool which I haven't tried. I have taken a few psnr, ssim, and vmaf measurements myself with ffmetrics, however. What catches my eye are the cases when their rankings disagree. My own eye seems to align with vmaf in those cases.

There are also tools available using FFmpeg which IMHO are not as accurate due to their usage of large block sizes in their computation. Also, if you use the RenderPlus tool in HOS, most codecs allow for calculation of both PSNR and SSIM as part of their output.

I've been saying this same thing since I ran my own comparisons against the MSU RQMT benchmark years ago. ffmpeg results are simply too top-heavy and undifferentiated by comparison. I use your non-ffmpeg tool almost exclusively. And VMAF is a marketing, not a measurement analysis tool.

Many thanks everyone I will look into it, I'll start with wwaag's tool as I'm trying to get a feel for using Happy Otter tools.

The happy Otter free tool seems to use cumin code which wants me to buy a licence at £25. I have managed to try a trial of cumin and get this:

Magix HEVC 5/10 MSE = 175, PSNR = 26

Magix HEVC 28/50 MSE = 62.092, PSNR = 30.446

Magix HEVC 50/100 MSE = 43.059, PSNR = 32.04

Magix AVC 28/50 MSE = 71.143, PSNR = 25.329

Magix AVC 50/100 MSE = 46.89, PSNR = 31.71

Voukoder HEVC 28/50 MSE = 59.829,PSNR = 30.653

@andyrpsmith

The Vegas Render Quality (VRQ) free tool can use either the newer CuminCode FrameServer(CCFS) or the older DebugMode FrameServer (DMFS). If CCFS is not available, it uses the older DMFS. A 30 day trial version of CCFS was included in the latest build of HOS (1.0.3.22). CCFS is required only for the latest build of V19 and V20. A CCFS license may be purchased on the HOS website for $15 US.

"How do you get the RQ launch from Otter scripts to find the RQM exe?"

Not sure exactly what you mean by this. You can add the VRQ script (or any script for that matter) to one of the OtterBars by right-clicking on an empty space and selecting "Add Script Action". Here's an example of VRQ being added to OtterInfoBar which I did for testing.

One important thing. When doing VRQ testing, it is recommended that the project pixel format be set to Legacy 8-bit (video levels). Once a rendered clip is added back to the timeline, make sure that the waveforms using VideoScopes are roughly the same for both the rendered and original clip.

Many thanks Wayne. Because I am using V20 it means I have to only use CCFS? Is CCFS also used in other happy Otter functions? I have been using full range will I need to recalculate everything? I must have fixed the finding VRQ as it comes up on its own now. What is a good value for Mean Square error (0 perfect?) I am getting 62 for my HEVC template that looks great.

@andyrpsmith

In my view, CCFS is almost "essential" for HOS in V20. From the change log:

Tools Affected: Render+, KwikPreview, RenderEvents, ProxyAssist, FrameTools, AviDub, and
DeshakerRedux

An MSE = 0 would be perfect. I've only seen it with lossless codecs such as magicYUV.

PSNR and MSE are very similar in that they use standard regression math to calculate the signal-to-error ratio. They are not perceptually weighted, so they are of limited use. Generally, a PSNR of 40 or more is an acceptable signal-to-noise ratio for second-generation video, with 100 being no noise (0 MSE).

The perceptually-weighted metric is SSIM, and it works more like our eyes. It takes temporal (motion) differences into account, as well as spatial domain noise. It is the far better metric for evaluating motion video, and the scale is 0-1.000, with 1.000 being a theoretical perfect match. In practice, optimal quality is usually reached around 0.995 SSIM, in order to prevent runaway bitrates and file sizes much larger than the original.

Glad to see you are taking an interest in this stuff. Our eyes are the least reliable visual "metric", they being moving targets themselves, and subject to all manner of tomfoolery, including ambient light, time of day, circadian rhythm, fatigue, blood chemistry and sugars, and our all-governing personal expectations (make no mistake -- the observer is the biggest part of the equation).

@andyrpsmith

In my view, CCFS is almost "essential" for HOS in V20. From the change log:

Tools Affected: Render+, KwikPreview, RenderEvents, ProxyAssist, FrameTools, AviDub, and
DeshakerRedux

An MSE = 0 would be perfect. I've only seen it with lossless codecs such as magicYUV.

Hi Wayne CCFS is still asking for an activation code, I followed the go into HO and activate, takes me to the HO site and I complete the form. Not sure what happens next. Do you have to activate it at your end? Many thanks.

PSNR and MSE are very similar in that they use standard regression math to calculate the signal-to-error ratio. They are not perceptually weighted, so they are of limited use. Generally, a PSNR of 40 or less is an acceptable signal-to-noise ratio for second-generation video.

The perceptually-weighted metric is SSIM, and it works more like our eyes. It takes temporal (motion) differences into account, as well as spatial domain noise. It is the far better metric for evaluating motion video, and the scale is 0-1.000, with 1.000 being a theoretical perfect match. In practice, optimal quality is usually reached around 0.995 SSIM, in order to prevent runaway bitrates and file sizes much larger than the original.

Glad to see you are taking an interest in this stuff. Our eyes are the least reliable visual "metric", they being moving targets themselves, and subject to all manner of tomfoolery, including ambient light, time of day, circadian rhythm, fatigue, blood chemistry and sugars, and our all-governing personal expectations (make no mistake -- the observer is the biggest part of the equation).

Thanks for the info, it is very complex. Is the best metric in the HO process SSIM I have not calculated it as it takes a long time if done together with PSNR.

 Is the best metric in the HO process SSIM

Yes. You don't have to run it for the whole video; a representative motion scene will do nicely.

Running it now on one clip and it is on 81 of 258 after ten mins.

CAUTION: When running any kind of render quality metric, make absolutely sure the first, last, and total frame count line up precisely and are identical.

REASON: Many encoders skip one or two frames at the beginning. Even a 1-frame mismatch will contaminate your results by up to 40%! The internet is filled with "revelations" by idiots who have ignored this most basic precaution.

Stopped it and restarted with select every 5th frame instead of 1 gave the following for HEVC 28/50

MSE 61, PSNR 30.5, SSIM 0.2614

SSIM 0.2614

I think something's wrong. See post above.

I agree. I've never seen an ssim number that low. Lowest numbers I've ever gotten are: psnr 23.9758, ssim 0.7217, and vmaf 43.7249. And that's comparing to the most demanding lossless footage of the ocean that I could dream up. Lowest 4k render bitrate I've ever bothered to test was 10.7 mbps. If your source is 100mbps avc, I'd start hevc at 80mbps and evaluate my way down from there.

Btw, I really like your visual artifact identification and counting approach to quality assessment. I can envision a similar automated approach using fournier matrix analysis to locate and count unnatural anomalies introduced into output, weighted for perceptibility and context.

I agree. I've never seen an ssim number that low. Lowest numbers I've ever gotten are: psnr 23.9758, ssim 0.7217, and vmaf 43.7249. And that's comparing to the most demanding lossless footage of the ocean that I could dream up. Lowest 4k bitrate I've ever bothered to test was 10.7 mbps. If your source is 100mbps avc, I'd start hevc at 80mbps and evaluate my way down from there.

I have been trying various bit rates with both AVC and HEVC and comparing how they look on the TV. Also trying to find a reasonable file size. My video is normal 5-10 mins. From what i can see so far HEVC is following the expected visuals of maintaining quality (by looking for blocking/artefacts) at lower bit rates (and therefore file size) than using AVC. I could run the template at an average bit rate of 80 mbps but from what I can see HEVC at 28mbps ave 50 max is showing a great image with no visible issues on my fine detail with movement clip. My camera can also run at 80mbps but even though I have never seen any difference to 100 I have always recorded at 100.

Btw, I really like your visual artifact identification and counting approach to quality assessment. I can envision a similar automated approach using fournier matrix analysis to locate and count unnatural anomalies introduced into output, weighted for perceptibility and context.

In one clip my son walks by in a green t-shirt which goes blocky really quickly as the bit rate drops also the fine shingle on the path goes nasty at lower bit rates.

but from what I can see HEVC at 28mbps ave 50 max is showing a great image with no visible issues on my fine detail with movement clip.

At 4K resolution? That's about Youtube download quality iirc, and if it's ok with you visually, you probably don't need to give it a lot more investigation.

@andyrpsmith

"I could run the template at an average bit rate of 80 mbps but from what I can see HEVC at 28mbps avg 50 max is showing a great image with no visible issues on my fine detail with movement clip."

Keep in mind that source video complexity drives the bit rate required when using Voukoder Constant Rate Factor.

Here are the file sizes for three 30 second UHD videos of different complexity rendered with the same settings.

CRF=22, GOP=30, GOP Minimum=15, No B Frames

Low Complexity

Medium Complexity

High Complexity