@Former user single VMAF is also arithmetic mean of all frames' values. Exact function (how the single value calculated based on flames values) does not really matter. It is very possible that two videos with very different characteristics will end up with quite close final values. I calculated VMAF for my synthetic example above and the VMAF value for the 1st video is better as well. But graphs shows much better story (I've added them to my first post).

I couldn't resist. This FHD table has the ffmpeg VMAF values added. Note that the table is divided into 4 sub-tables where sorting is only relevant to each sub-table, not the full table.

UPDATE: Relating to the following posts for ... "UTVideo", "QSV" and "MC Legacy CPU" items. This table has been now updated, thanks to fifonik in modifying ffmpeg syntax.

UTVideo is currently left out.

UPDATE 2021: The table is now the latest.

@JN- UTVideo's looks suspicious: for lossless the ffmpeg SSIM, ffmpeg PSNR & VMAF values are too low.

The two others that look suspicious are the QSV and MC Legacy. Another thing to note is that most FFmpeg SSIM values are extremely high. I wrote another version of RQM based on FFmpeg and finally gave up because of the inconsistent results. Since FFmpeg always compares two files, it seemed likely those "suspicious" results are the lack of the exact "synchronization" of the reference and test file. E.g. I would sometimes find huge differences between the original and a rendered file that was "lossless". If anyone would like to use the FFmpeg version, I'll make it available.

I should mention that HOS now has both PSNR and SSIM options for a number of render formats including x264, x265, Nvenc and a couple of others as I recall. Those calculations are computed during the render itself and represent the difference the "input" frame and "rendered" frame. Again, you see these very high SSIM values that seem artificially inflated for some reason. My suspicion is that they occur due to the way that those values are being computed--I've read that an 8x8 block size is being used (sorry I don't have the reference). The computations inside of RQ are based on each pixel and as such are probably more accurate, although the computation is much slower. If you increase the block size in RQ, computation is quicker and the SSIM values go up. It's a speed/accuracy tradeoff.

The advantage of the RQ implementation is that the difference calculations are done inside of Vegas and can be better controlled. It only ouputs the "difference" between the original and the rendered frame which results in a series of completely "black" frames if the render is truly lossless.

At the end of the day, it probably doesn't matter all that much since most users are completely happy with the "eyeball" approach to render quality, which at the outset of this thread, I stated is perfectly OK.

The two others that look suspicious are the QSV and MC Legacy. Another thing to note is that most FFmpeg SSIM values are extremely high.

Yup. However, for these files there are some possible variables and I cannot judge. For lossless it is just impossible to have so low value that is not aligned with the values calculated using different algorithms.

When I just started to play with VQMT I faced similar issue. At the end I found that some the AviSynth's source plugins were skipping a few frames at the very beginning that makes resulting numbers/graphs very inaccurate.

If files (reference & encoded) are available it would be possible to double check and may be pin-point the exact issue or suggest a workaround.

As for SSIM in ffmpeg... I agree, that it is implementation is really strange and inconsistent. I'm getting very different results when calculating SSIM/PSNR using different methods with the same ffmpeg:

ffmpef -hide_banner -i 1\1.mp4 -i ref\ref.mp4 -lavfi "ssim;[0:v][1:v]psnr" -f null -

=>

SSIM: 0.866613

PSNR: 14.749501

ffmpeg  -hide_banner -i 1\1.mp4 -i ref\ref.mp4 -lavfi libvmaf="model_path=vmaf_v0.6.1.pkl:log_fmt=json:psnr=1:ssim=1:log_path=1/ffmpeg_vmaf.json" -f null -

=>

SSIM: 0.853249654173851

PSNR: 35.86960090021684

UPDATE:

BTW, @wwaag have you tried to use setpts filter option to discard timestamp offset? In addition, I think eof_action=endall must be used to deal with files that have different lengths.

Something like this:

ffmpeg -hide_banner -i main.mp4 -i ref.mp4 -lavfi [0:v]setpts=PTS-STARTPTS[main];[1:v]setpts=PTS-STARTPTS[ref];[main][ref]psnr="eof_action=endall:stats_file=ffpsnr.log" -f null -

I generated the UTvideo sample from HOS, the QSV and Legacy MC are generated from within VP direct.

@fifonik Feel free to do a sample test using the three codecs mentioned above, looking forward to your results.

You know the drill, create a high quality source.mp4, generate a QSV and MC Legacy codec etc etc.

I've been aware for as long as I have been doing this testing that QSV and MC Legacy were poor enough. They have always been tracking with respect to the other codecs a little irregularly, but still consistently near the bottom of the pile, same for 4K UHD versions. You can see the link in my profile to the 4K and other tables. But the FHD results here in this post are very poor, abnormally so using ffmpeg, not so the UHD results using ffmpeg, and the HORQ results are also ok.

An eyeball playback check of these shows them ok. There has been plenty of mentions of QSV not been too good by other posters, but if your best HW encoding option available is QSV, and it looks ok, then fine.

I saw immediately the very poor FHD results for these 2 using ffmpeg VMAF. I suspect VMAF doesn't like them very much, neither does ffmpeg's SSIM and PSNR.

As for UTVideo, I only have it through HOS, but the thing is that it sorts into 2nd. position using HOS SSIM and it comes in above MagicYUV. These 2 are mathematically lossless, unlike the no. 1 position of Sony YUV, make what you will of that.

I feel it's important to observe that up to now, although poor performers, they do have some tracking consistency, within the HO RQ test results and therefore, beware of going down rabbit holes only because of the ffmpeg results.

What I did check for was any "levels" change in these 3 codecs, (similar to the abnormal one for Xavc-L in 4K) but found none, and no frame position differences, although I haven't checked all 675 frames!

Also ffmpeg did throw out this error message while processing the MC Legacy CPU FHD & UHD versions ...

"not matching timebases found between first input: 1/25 and second input 1/25000, results may be incorrect" Despite having this error message the UHD ffmpeg results are aok, This error message didn't occur for the QSV FHD test clip. Just saw that this message appears in quite a lot of ffmpeg processing of various clips, so I don’t think it’s pointing to an answer to this issue.

When one looks at the HO SSIM sort, I think that that is probably correct, i.e. the best column to sort on. The problem lies with the FHD ffmpeg results, not only of VMAF but also ffmpeg's results for SSIM and PSNR for these 3 codecs. This ffmpeg issue doesn't apply to the UHD QSV and MC Legacy Cpu test results.

The conclusion I have come to is that these three codecs are in their correct positions with respect to the HO SSIM sort and that the really low (FHD only, UHD is ok) ffmpeg results are a mystery so far.

If anyone would care to do a FHD test using these three codecs plus any other one of the codecs that does give consistent ffmpeg results ok. This would be to see if there is a similar ffmpeg FHD only test, discrepancy with these three. Obviously it would be best to use your own Source.mp4 file to remove any further confusion.

@wwaag

Is the RQM Tool version 2 in for repairs? The download page gives me version 1.

@Musicvid

You're probably downloading the wrong version. Version 2 is "Vegas Render Quality" under Vegas Script Extensions.

OIC. I downloaded the standalone which is V1.

I was also trying forgetfully to run V2 extension as a standalone which obviously doesn't work.

@fifonik

The standalone FFmpeg app I wrote uses the first command line. Seems to me that I tried the other command line as well, but finally gave up.

@JN-

Just ran a few tests just to make sure that RQ was working OK in V14 and included QSV in addition to straight x264 and Nvenc. For the clip I was using MSE for x264 was around 6. For QSV it jumped up to 30 and surprisingly Nvenc was around 4. Something seems to be definitely wrong with QSV.

@wwaag I’m trying to narrow this 3 codec ffmpeg results issue down. I suspect that UTVideo being a mathematically lossless codec may be as good as it gets, perhaps simply no problem.

I'm for the moment concentrating on one of the two other problem codecs, MC Legacy CPU. I'm going with the assumption that the HO RQ results are ok and it’s in my testing with ffmpeg that the issue is, whether it’s because I need to have a better ffmpeg syntax or it’s VP rendering out anomaly’s.

In further testing yesterday I found that the ffmpeg poor results simply go away above a certain data rate.

The abnormally low ffmpeg results stay at the low value until a specific FHD data rate cutoff point. The results are what you would expect after and above that cutoff point, but its a very high data rate for FHD. This didn’t show up as an issue in my UHD testing because my target data rate for UHD was 135/100 in the render templates.

Cutoff data rate point, before and after. VP render template data rates ... before=poor=low SSIM/PSNR results, 88,093,137 and after=good=high SSIM/PSNR results, 88,093,138 data rates. The max in both cases was set the same at 135. The in table in use item 24 MC Legacy render settings obviously falls into the “before” section at ~ 20,000,000 data rate.

The before and after ffmpeg SSIM (DB) results for these was 5.859504 and 23.362238.

The before and after ffmpeg VMAF results were 10.817757 and 99.566025

When I put these two rendered out clips one above the other in vp and apply a “difference”, on playback in some places the histogram changes from a rock solid very narrow line on the left side to a little wider.

Its important to remember that despite these visual differences in the histogram, before and after, that the HORQ results are acceptable for the lower data rate, in use item 24 MC Legacy cpu, but not ffmpegs.

@wwaag This cutoff issue also applies to the QSV VP rendered files. The following are the values associated with the two files before and after the cutoffs, for both item no. 22 and item no. 24. So a total of 4 files, not including the original rendered file for 22 and 24 both ~ 20 Mbps.

"Before" is a bad ffmpeg result and "After" is a correct ffmpeg result.

ffmpeg before and after values surrounding the cut off point for item 24 MC Legacy Cpu ...

The render template data rate cut-off is between 88,093,137 and 88,093,138.

In other words anything => 88,093,138 is ok and anything =< 88,093,137 is not ok.

Before ... SSIM All .. 0.740552 (5.859504) ..... PSNR Avg. .. 21.723365 ..... VMAF .. 10.817757 ..... Size 286 Mb ..... Data rate .. 88,864 kbps
After .... SSIM All .. 0.995389 (23.362238) ..... PSNR Avg. .. 48.361580 ..... VMAF .. 99.566025 ..... Size 286 Mb .....  Data rate .. 88,879 kbps

ffmpeg before and after values surrounding the cut off point for item 22 QSV ...

The render template data rate cut-off is between 71,151,999 and 71,152,000.

In other words anything => 71,151,999 is ok and anything =< 71,152,000 is not ok.

Before ... SSIM All .. 0.739620 (5.843929) ..... PSNR Avg. .. 21.715071 ..... VMAF .. 10.930863 ..... Size 227 Mb ..... Data rate .. 70,570 kbps
After .... SSIM All .. 0.989100 (19.625552) ..... PSNR Avg. .. 44.501228 ..... VMAF .. 99.392126 ..... Size 227 Mb .....  Data rate .. 70,575 kbps

Given this issue with ffmpeg, but only with the Qsv and MC Legacy codecs, and only with FHD, I’ll just leave the ffmpeg entries blank for those two items, currently no. 22 and no. 24, when I update the FHD table again. The HORQ entries are valid as they track with respect to what I have already in the UHD tables.

It would be possible to generate a few other codecs in FHD to go with these two at the higher “After” data rates but it’s really unnecessary since the primary point of the exercise is to get some idea of pecking order, and that’s pretty much established.

I guess your uprezzing thread had been siderezzed quite a bit with all of this🤪. Maybe it’ll get back on track soon.

I think I spot the "issue" you have while calculating these metrics. In files you sent me 22-* files have 675 frames, while 24-* files -- 676 frames (ffprobe and virtual dub 2 both show that). Media info reports different duration for these files: 27.0 secs for 22-* and 27.40 for 24-*. This is likely affects the metrics.

I've used my ffmpeg options (with setpts and eof_action, see above) to calculate SSIM/PSNR metrics and they are not as low as they used to be in your results. I've sent you scripts I used and results I got (check PM).

Hope this help.

@fifonik As PM, I found that VP shows 675 frames for both files, same as source. HO duplicates frame finder finds a single frame a bit from the end, in both 22 and 24, but in both cases its found within a still image within the video clip, so it may be incorrectly reporting. It doesn’t find any duplicate at the end of clips.

I'm not sure how the HO tool works, whether it’s just comparing frame contents and can be fooled in this way with a static section of video, or whether say timestamps are involved and so it is a real duplicate frame.

@fifonik Here's the thing, as you say Mediainfo reports a different/longer (27.4s) duration for the 24 QSV original rendered out file, it should be 27.0s. So QSV, HW encoding may be adding an extra frame that's not visible in VP, weird. The no. 22 MC Legacy, CPU encoded file has the correct duration.

Nvenc HW encoding also adds a duplicate frame (is long time ago added to Nicks known issues list), for 29.97 fps 4K UHD.

But both file no. 22 and 24 have a similar issue when getting SSIM, PSNR and VMAF ffmpeg values, not so with HORQ SSIM and PSNR results.

So although this is a new discovery, it doesn't explain the ffmpeg results for 2 reasons.

1) Number 22 QSV file has no duration change and ffmpeg still gives it a similarily abnormally low result, HORQ tests don't give it a low result like ffmpeg.

2) The low ffmpeg results problem for number 24 disappears once you go above the data rate cut off point. Same for number 22.

It's important to take into account all 7 files involved in this.

1) Source.mp4 ... 27s ... used to generate/render all 6 files below.

2) 22 QSV ~ 20 Mbps.mp4 ... has an abnormally low ffmpeg result, but HORQ is ok

3) 22-71-151-999-Before.mp4 ... has an abnormally low ffmpeg result, but HORQ is ok

4) 22-71-152-000-After.mp4 ... has a correct ffmpeg result, HORQ has also

5) 24 MC Legacy ~ 20 Mbps.mp4 ... has an abnormally low ffmpeg result, but HORQ is ok

6) 24-88-093-137-Before.mp4 ... has an abnormally low ffmpeg result, but HORQ is ok

7) 24-88-093-138-After.mp4 ... has a correct ffmpeg result, HORQ has also

The numbers in the filenames of 22 and 24 are the exact render template entries used to generate them, the Max was set to 135 for 3), 4) 6) and 7). For example 4) is 135/71,152,000. So entering this data rate value in the render template means that the ffmpeg issue goes away for file “22-71-152-000-After.mp4"

That's no good to me as I need to render it only to ~ 20 Mbps, not ~ 70.5 Mbps as in the case of 4).

@fifonik PM

I also think that there is a timestamp issue and as a result in one file frames are shifted for ffmpeg point of view. 1 frame from first file compared to 2 frame from the second file etc.This is why adding the setpts option is fixing the issue.

One additional duplicated frame at the end (if any) should not change the total metric value so dramatically. Default eof_action action in ffmpeg is duplicating the last frame so if in longer file the last frame is duplicated (this is how it looked in VDub2 when I checked) this should not affect the metric values at all as in short file the last frame would be duplicated as well.

In practice, with mentioned options added, PSNR/SSIM metrics for "22-71-152-000-After.mp4" have not changed (43.510915/0.990605), and for "22-71-151-999-Before.mp4" they were fixed (19.983446/0.632726 => 43.538394/0.990667). New values do not look too low.

@fifonik I did several tests on the number 22 and 24 files and your syntax change has certainly fixed it, thank you. I checked some existing files without issues and everything AOK, same results as previous.

Just the VMAF to go :D.

See PM.

@Musicvid @wwaag @fifonik I've now update the table posted earlier with the modified values, thanks to fifonik's ffmpeg syntax changes in fixing same, couldn't have done it without his help.

The UTVideo issue was resolved by simply rendering out a new piece from HOS, it was slightly different in size and data rate to the UTVideo file that I earlier rendered out from HOS. I guess I picked the wrong one initially from within HOS.

I've taken a screenshot of the one that's ok, not sure which one I used previously, but the Mediainfo data for both is identical.

The odd thing is that the ffmpeg values for UTVideo now and Magix Intermediate Lossless are identical, coincidence I guess, because I double checked I was testing the correct files.

Forgot to do HOS RQM for the new UTVideo, item number 2, will update later.

Later: Well that was a terrible waste of time. I've removed item 02 Utvideo from the table. When I did get good results with ffmpeg they were very bad with HORQ. There are 4 render options within HOS for UTVideo, the 10 bit 422 file wouldn't load into VP, the other three were obviously poor just using the "difference". I came to testing all 4 options after doing a full RQM test using the best ffmpeg result item, one of these, probably the first. Namely UTVideo YUV420 BT709 VCM & UTVideo YUV420 BT709 VCM - YV input.   These two give very good ffmpeg results, but all three that would load into VP17 were obviously poor using "Difference" ...

ffmpeg results ... SSIM All 0.998766 (29.085636) ... PSNR 56.568659 ... VMAF 99.750513 for both.

I also tried rendering out directly from VP, not using HOS, there were extra UTVideo render options including "Pro", but it said missing codec for the Pro, black screen.

The UTVideo that I had rendered out previously, some time ago, was probably done direct from VP, because it was ok for HORQ metrics, but poor for ffmpeg, so no go.

So to summarise the problem with UTVideo codec, whether the rendered out files were from within HOS or direct from within VP in vfw. I tried several, 4 from within HOS and at least 4 from within VP vfw direct.

If the ffmpeg results are good, then the HORQ results are bad.

If the HORQ results are good, then the ffmpeg results are bad.

I never get any case where the ffmpeg and HORQ results are both good for a single clip.

Of the 26 other test codecs in the FHD table none behaved like this, there was always a consistency between the ffmpeg and HORQ results. Where there was recently with items 22 and 24, thats been fixed.

So I think theres no point in leaving the UTVideo results in the table as it’s giving an unclear result.

I don’t think many use this codec anyway, if they do maybe they should reconsider given these flaky results.

Meant to say that after doing the first full HORQ test, with poor results, I only needed to use the “difference” check in VP to quickly establish if the file was ok or not, this saved some time. Obviously if I had got both tests good I would then have done full HORQ tests to get the exact values.

When I did the difference testing I could see that for all of the samples that failed this test but succeeded the ffmpeg tests that the levels were being expanded by ~ 15, checked at 5 different points on the left. Example source = 28 but rendered = 13. Similar is happening to the right but can't be measured accurately because the source was mostly near 255.

When I did the difference testing I could see that for all of the samples that failed the ffmpeg test but succeeded in this test that the levels stayed the same

Some of the test results ...

UTVideo ...

These are test files rendered out to see if I could get even one that gave both good ffmpeg and good HORQ results, but none did.

Some information on the UTVideo poor HORQ and ffmpeg results.

Files can be output from within HOS or direct from within VP under "Video for Windows codecs".

It wasn’t necessary to do a full HORQ metrics test as the “differencing” test within VP shows the issue. If I had had a good differencing result with a good ffmpeg result I would have done so.

HOS ... #1 and #3 appear to be near identical.

Number #1-YUV420 BT709 VCM.avi
Number #2-YUV420 BT709 RGB Input.avi
Number #3-YUV420 BT709 YV Input.avi
Number #4-422 10 bit BT709.avi (VP cannot load this file)

VP direct, VFW ...
Number #5-YUV420 BT709 VCM.avi 
Number #6-T2 YUV422 BT709 VCM.avi

Results ...

#1 HORQ = Bad   ffmpeg = Good ..... ffmpeg SSIM All .. 0.998766 (29.085636) .. PSNR Avg. .. 56.568659 .. VMAF .. 99.750513 

#2 HORQ = Bad   ffmpeg = Bad ...... ffmpeg SSIM All .. 0.995794 (23.760805) .. PSNR Avg. .. 39.577188 .. VMAF .. 97.715213

#3 HORQ = Bad   ffmpeg = Good ..... ffmpeg SSIM All .. 0.998766 (29.085636) .. PSNR Avg. .. 56.568659 .. VMAF .. 99.750513

#4 HORQ = N/A   ffmpeg = Good ..... ffmpeg SSIM All .. 0.997203 (25.533755) .. PSNR Avg. .. 53.262434 .. VMAF .. 99.750513

#5 HORQ = Bad   ffmpeg = Bad  ..... ffmpeg SSIM All .. 0.991657 (20.787011) .. PSNR Avg. .. 31.289519 .. VMAF .. 71.302051

#6 HORQ = Good  ffmpeg = Bad  ..... ffmpeg SSIM All .. 0.991088 (20.500367) .. PSNR Avg. .. 31.280647 .. VMAF .. 71.302051

Where the HORQ results are "Bad" the "levels" within VP are showing to be expanded by ~15.  So a levels of say 28 on the source file becomes instead 13. 

By way of comparison the results for "MagicYUV were ...  HORQ = Good ... ffmpeg = SSIM All .. 0.998402 (27.964279) .. PSNR Avg. .. 55.023211 .. VMAF .. 99.750834

@JN-

Could you make your 7 test files available and also show exactly what FFmpeg command lines you are using?

I've added the Reset Time Stamp option to my FFmpeg version and will upload to the HOS website later today. Here's the screen grab.

The two command lines in use internally are the same as @fifonik posted above--with and without resetting the timestamp.

The wildly different results you see are the reason I abandoned use of FFmpeg. If you search, you will find that others have reported similar misgivings in the use of FFmpeg for such calculations. I think there are two major reasons as it pertains to Vegas.

First, there are differences in decoding. Vegas decodes yuv and rgb formats quite differently. In fact, one of the major concerns in HOS is rendering out from Vegas, performing some filtering option, re-rendering and maintaining levels once the rendered file is added back to the timeline. It's doable, but a challenge that requires different conversions for different codecs. The advantage of using RQ in Vegas is that you can immediately see if there are differences by looking at the scopes. You simply can't do this in FFmpeg.

Second, there are differences in accuracy. In order to achieve speed (you can see the fps in the FFmpeg version) of 100+ fps, you have to increase the block size that is being evaluated. That's why the HORQ version chugs along at just a few frames per second--it looks at every pixel. If you increase the block size of the initial render from Vegas (actually resizing to a smaller frame size), you will see a systematic increases in PSNR and SSIM as the block size is increased.

As an alternative to FFmpeg, you could also make use of the Avisynth compare function http://avisynth.nl/index.php/Compare which has some decided advantages in my view. If I were to continue any further effort in RQ, it would definitely be in the use of Avisynth which offers its own set of challenges. The bottom line is that there are lots of options for one to pursue.