Did a little testing with my own reference file, which is a 4K hevc-lossless raw recording made with a zcam E2. Was curious about the frame-length discrepancy mentioned here earlier. Indeed, the reference clip is 921 frames while all the Vegas renders, no matter which encoder or decoder is used, are 920 frames. Also did test renders with ffmpeg, both with and without hardware encoding and decoding, and they were all exactly 921 frames. Don't know how this impacts ffmetrics numbers but it might explain why all of the Vegas numbers are lower in comparison to ffmpeg. Btw, I used ffprobe packet counting to determine frame counts. Here's a short summary of my results:

Btw, I wasn't able to duplicate the ffmpeg results with Voukoder in Vegas because I don't know how to match ffmpeg options to Voukoder menu choices. Although I'm sure there's a way to do it. Also, it might not be possible to hook ffmpeg hardware encoding into Vegas. In any event the ffmpeg hardware numbers are quite spectacular... be nice if Vegas just wired in their libs like Resolve does.

I mentioned an idea with re-enoding a file 10x, that is constantly replacing the clip on timeline with the new encode. Using MagixHEVC software encode with SO4 decoder and then again with Legacy decoder

By the 10th encode the video starts on frame30 using Vegas with SO4 decoder, this file starts at frame 27 when brought onto timeline using Vegas Legacy decoder. It starts frame27 using media players

Obviously I can't plug these encodes into a software analysis tool.

Anyway the point of this was to compare original with encode 10. encode 10 looks horrible, so currently I'm of the belief the SO4 decoder is low quality in playback, and it will cut the first 2-3 frames, by the 10th encode the audio sync is out by 30frames (SO4 decoder), and 30 frames shorter. I will have to do the same with Legacy decoder to determine if SO4 is really to blame or if it's the process of re-encoding a file 10x even with software encoding

This is Encode 10 (SO4 decoder, MainConcept HEVC (default)

*(The forum won't let me post pictures, I'll chk if that's possible later via an edit)

You should be able to post pics using the upload button next to the emoji button, but only some formats are supported.

Did a little testing with my own reference file, which is a 4K hevc-lossless raw recording made with a zcam E2. Was curious about the frame-length discrepancy mentioned here earlier. Indeed, the reference clip is 921 frames while all the Vegas renders, no matter which encoder or decoder is used, are 920 frames. Also did test renders with ffmpeg, both with and without hardware encoding and decoding, and they were all exactly 921 frames. Don't know how this impacts ffmetrics numbers but it might explain why all of the Vegas numbers are lower in comparison to ffmpeg. Btw, I used ffprobe packet counting to determine frame counts. Here's a short summary of my results:

Btw, I wasn't able to duplicate the ffmpeg results with Voukoder in Vegas because I don't know how to match ffmpeg options to Voukoder menu choices. Although I'm sure there's a way to do it. Also, it might not be possible to hook ffmpeg hardware encoding into Vegas. In any event the ffmpeg hardware numbers are quite spectacular... be nice if Vegas just wired in their libs like Resolve does.

@Howard-Vigorita My understanding at this point is that the size change, or frame loss, happens when placing the file on the timeline, not at the time of render. So if you were reloading a project with your reference clip already on the timeline, the size of the render is probably fixed according to the configuration of the project at the time it was created. So renders would all be the same size. Not 100% sure about this. There were too many unexplained size changes when I was first working on this. Also, I actually noticed size changes of fractional frames.

I gave up and compared the same frame visually rather than using software analysis of the videos. The number on the file name is the generation, so 10 signifies 10th generation encode. I don't see the quality loss of SO4 that was expected, but it is only comparing a single frame and not a video.

I guess this is more a review of the MagixHEVC software encoder. Are these sort of quality loses expected or unusual?

*EDIT: I'm no longer able to upload any media to this forum, if you're feeling brave you can see the picture comparison at this 3rd party link https://cdn.discordapp.com/attachments/774137864501067790/996310130046013520/testing.png

Don't know why you are having a problem but anyway I've added it for you (EricLNZ - Mod).

Thanks for that EricLNZ, I see in both cases the picture loses too much quality to be of much use.

If anyone wants a clear look download this self contained web archive with the original pictures, double click to open in your default browser . Go to split screen view, the left side is the reference image, you choose the image you want on the right by clicking on the image selection on bottom right. Move the slider left and right to compare

You might see something interesting, in this test Legacy HEVC looks terrible at encode1, while SO4 looks much better. So this has been a confusing affair

https://www.dropbox.com/s/f3ex4ne4sk2xjxu/compare.html?dl=0

My understanding at this point is that the size change, or frame loss, happens when placing the file on the timeline, not at the time of render. So if you were reloading a project with your reference clip already on the timeline, the size of the render is probably fixed according to the configuration of the project at the time it was created. So renders would all be the same size.

@bvideo Don't know either but I think you're right. I just recreated the test project from scratch with the most recent build of vp19, build 643, and all my renders are now adding 1 frame... I'm now getting 922 frame renders. No matter which render or decoding codec is used. But the 1st 90 frames still come up screwy in ffmetrics charts. However, the end numbers only alter slightly from before. Here's one of the vmaf charts I got with legacy hevc decoding and nvenc render which yielded 920 frames:

With the recreated project, same settings I got 922 frames and vmaf came out slightly better (95.5490 vrs 95.5462) but the same low screwy results in the 1st 90 frames.

Contrast these with the vmaf chart (vmaf score: 98.7872) of the ffmpeg render using qsvdec & nvenc that yelded the correct 921 frames... some irregularity at the beginning, probably a consequence of the way ffmpeg is doing rolling averages, but not nearly as extreme in the beginning as from Vegas. But Vegas looks about the same as ffmpeg once it gets going:

edit: Just did a skip of the 1st 5 sec in ffmetrics doing the vmaf chart of the same Vegas render as above and it looks similar to the chart of the ffmpeg render. The irregularity is limited to about the 1st 25 frames. Convinces me that the first 25 frames of irregularity is something computational in ffmetrics/ffmpeg. The Vegas render's vmaf number jumps from 95.5490 to 97.5325... much closer to, but still below the overall score of 98.7872 from the ffmpeg render.

Btw, if anyone wants to duplicate my methodology, here's the ffprobe command I used to count frames... packets actually, but the result is the same, just quicker (substitute 'frames' for 'packets' to verify... but wait for it):

ffprobe -v error -select_streams v:0 -count_packets -show_entries stream=nb_read_packets YOURCLIP.MOV

Here's the ffmpeg command I used to do a qsv decode with an nvidia encode... I forced these to 240mbps to match what I did in Vegas renders:

ffmpeg -y -hwaccel qsv -i YOURCLIP.mov -c:v hevc_nvenc -b:v 240M -minrate 240M -maxrate 240M -bufsize 480M d:\temp\ffm_v19_ffmpeg-nvenc-qsvdec.mp4

Here's the similar command I used for nvidia decoding and qsv for encoding:

ffmpeg -y -hwaccel cuda -i YOURCLIP.mov -c:v hevc_qsv -b:v 240M -minrate 240M -maxrate 240M -bufsize 480M d:\temp\ffm_v19_ffmpeg-qsvenc-nvdec.mp4

And the command for x265 software encode/decode:

ffmpeg -y -i YOURCLIP.mov -c:v libx265 -b:v 240M -minrate 240M -maxrate 240M -bufsize 480M d:\temp\ffm_v19_libx265.mp4

@Howard-Vigorita - I remain in awe of your abilities! I’m not too dumb to get a tiny handle on just what you’re doing. Stunning work, really. TeamVegas are fortunate to have you on the Forums.

@Howard-Vigorita I was using John Dennis test file, and it could serve you better. Each frame has frame and timecode text. If any software is not comparing identical frames it should be more obvious then a regular 1 shot video where a frame preceding or advanced can be very similar.

Like Grazie I no longer follow what you're doing, but I hope the insight you've formed will make it possible for you to create files that can give a meaningful result and answer if the HEVC decoder used makes a difference as far as the quality of an encode

I'm convinced now it's not the GPU decoding that is causing the HEVC frame count problem. It is the software that assesses source files on the timeline. By that, I think I mean SO4 as it is seeing an HEVC file loading or already on the timeline. If this is true, then configuring HEVC for nvdec probably has the same problem as QSV decode.

As a result, any attempt to compare HEVC decode quality using Vegas renders is doomed (or, generally, comparing render quality with HEVC sources). (As a workaround, someone could probably manually adjust the HEVC source file before render to appear identically on the timeline between the HEVC legacy and not legacy versions of the project.) AVC files don't exhibit this problem as far as I can tell.

For another wrinkle, I created and saved a project with my HEVC file selecting the HEVC legacy decoder. Then opened it with legacy unchecked. The event now had the original correct length, but it was missing the first two frames and was extended with two extra frames at the end to make up the loss. (Zooming in, you can see the notch.) This would cause the kind of glitch where frames from earlier in the project could appear duplicated at event boundaries, or black frames could occur. The Edit->switches choice for looping/holding/trimming determines which frames get duplicated. Changing that switch to "trimming" shortens the event which will either shorten the project accordingly or create blank frames, depending on ripple.

Conversely, creating the project with legacy unchecked, then opening it with legacy checked makes the project appear a little different, as if by slipping the shortened event to show the first two frames and hide the last two frames. In this case changing Edit->switches to "trimming" lengthens the events on the timeline and can lengthen the project and/or create or lengthen overlaps, depending on how overlaps & ripple are configured.

All of this is saying that changing the choice of HEVC legacy or not for an existing project can change subtle mysterious things about the project that have nothing to do with rendering.

I was using John Dennis test file, and it could serve you better. Each frame has frame and timecode text. If any software is not comparing identical frames it should be more obvious then a regular 1 shot video where a frame preceding or advanced can be very similar.

Like Grazie I no longer follow what you're doing, but I hope the insight you've formed will make it possible for you to create files that can give a meaningful result and answer if the HEVC decoder used makes a difference as far as the quality of an encode

@Former user I think that clip with timecode shown on each frame is exactly what's needed to diagnose if Vegas drops or adds frames or shifts them.

But, not to throw a monkey wrench into things, my laptop got updates to win11 and the Intel graphics driver this morning and now vp19 is yielding renders exactly on the money: 921 frames. The ffmetrics score and chart are exactly the same as indicated above when vp19 generated 922 frames yesterday... so apparently ffmetrics ignored that extra frame yesterday. Leads me to conclude the frame drop/add anomaly has more to do with the Windows file system or libs than Vegas. In any event, I'll probably run the same measurements again and add to my tables after vp20 comes out to reflect the minor improvement.

Here is how I've done my comparisons ever since I was using MSU and HOS QMT, the pre-ffmpeg version..

Disclaimer: I have long since abandoned trusting any ffmpeg quality measurements, after confirming the lack of upper end output differentiation, which is best described as being top heavy.

I put both comparison files on the timeline, trim head and tail frames until they exactly match up, and then render losslessly for feeding to QMT. This rarely results in initial output garbage results past the first few frames.

Trying to sidestep the issue of lost frames for HEVC files on the timeline, I wanted to see if there was any way to compare decoders using Vegas renders.

I did HEVC -> uncompressed renders for both QSV and legacy HEVC for three different HEVC files. But before each legacy render, I trimmed the events so that they matched the frames seen in the QSV project for that source. Each HEVC source I tried had a different amount of frame loss. My HEVC file, created in Vegas, lost two frames at the beginning. The B1010018.MOV file from the first HEVC sample referenced by @john_dennis lost one frame at the beginning. The Reference_H.265.mp4 file from the "known quantity" HEVC source referenced by John lost no frames, so no trimming.

After I rendered each of the three to uncompressed using both legacy and QSV, I ran ffmetrics comparing the pairs of uncompressed renders for each. They all reported inf and 1.0000 for psnr and ssim, but 99.xxx for VMAF. I have no idea what this means for VMAF. I could not find any differences viewing the frames in Vegas that were called out by VMAF. In the case of my file that has VMAF disturbances around frames 140-165, my file has a hard cut at frame 148, so I think VMAF is sensitive to scene jumps more than content differences.

It was not my intention to measure decoding quality per se because I was not comparing them with an original, just with each other, so just looking for differences. And not my intention to compare encoders at all, just decoders.

My file:

The two others:

Before I started this post, I was wondering why anyone would say one decoder could be better / worse / different from another. Now it appears that the actual decoding resolves perfectly according to PSNR and SSIM, just not according to VMAF (investigation needed). The problem of selecting a decoder in Vegas is that it is unpredictable what portion of a file will actually show up on the timeline with a GPU decoder before any decoding gets done.

Edit: It's also worth repeating that changing a project's options between legacy HEVC and SO4 with HEVC files on the timeline runs the risk of glitches and event slipping.

You're the best contributor to your own investigation, @bvideo, and thanks for understanding and embracing rule-out theory in your tests, as opposed to imposing your preferred spin on the data, which I might mention is popular sport here.

Your singling out of VMAF for further investigation may be partially due to its overall subjective metric, which is not imposed on either SSIM, PSNR, or RMSE -- the former having some perceptual weighting, the latter being pure regression math..

I still have seen nothing compelling enough to want to test HEVC decoders; either the asset opens or it doesn't. I guess I might if I absolutely considered the first or last couple of frames to be nonexpendable.

 

It was not my intention to measure decoding quality per se because I was not comparing them with an original, just with each other, so just looking for differences. And not my intention to compare encoders at all, just decoders.

I did the same, encoded with voukoder using SO4 and Legacy HEVC decode. This is my result

These results looked so suspicious to me, I had to do the trancodes again, this time checking GPU DECODE in taskmanager during the encodes. They are 2 different encodes, using the different decoders and they are identical according to software analysis

They all reported inf and 1.0000 for psnr and ssim, but 99.xxx for VMAF. I have no idea what this means for VMAF.

Yes, bvideo, since PSNR and SSIM use regression analysis, those results convey mathematically lossless results.

I have no idea what this means for VMAF.

Yes, VMAF uses a subjective metric, whose algorithm is based on fuzzy logic. For that reason, I avoid it entirely when making objective comparisons.  I'm still not sure where it crept into this discussion; I don't really care. VMAF was commercially developed by Netflix as a market centered tool for lossy streaming of consumer entertainment. My grandfather would say, "use the right tool for the right job."

If you were to use the RMSE, or Root Mean Square Regression in Wayne's HOS QMT, which is identical to a function on your TI84 calculator, it would give further validation of the results you quoted above.

You certainly have a clear head for this stuff, I hope you continue to contribute.

Although math and computation are kinda my thing, I know and care nothing about the math behind these metrics. But I look at the renders I do. The vmaf and ssim results best match my subjective evaluations. Particularly my impression that as bitrate and frame rate go up, that renders look better to my eye. Psnr seems to go the other way many times. I also note that ffmeg has switches to "tune" renders for psnr or ssim but I don't see the difference. Suggesting to me that those ratings can be fudged at will. I like vmaf the best because it gives a better numeric spread than ssim and, in the few instances their rankings disagree, the vmaf one matches my own. And I respect the fact that after using psnr and then switching to ssim, Netflix developed vmaf itself to better tune their own transmission compression to customer satisfaction. Only downside in my mind is vmaf takes longer to calculate. But I collect all 3 measurements from ffmetrics, allow each table column to be sorted, and each to their own.

@Howard-Vigorita

"Although math and computation are kinda my thing"

We appreciate that.

I still have some of my college Calculus homework. Of all the marks on the page, I recognize my handwriting.

An absurd extension of quantum mechanics theory suggests that if you give an engineer and a lawyer the same data, they will never draw parallel conclusions. Can I say that here?