@MagicYUV "and just right off the bat it's range is problematic." I intentially left it as is from SOOC sourced clips, not something I would do in a non test project. This lack of conforming the source clip has already proved useful in discovering an issue with one of the VP codecs (maybe XAVC-L ?) where VP render truncates, clips, it to limited in UHD but not in FHD. Marco confirmed this with his testing.

Where the flag is not reliable as in this case, and you are relying on it to process accordingly, then I suppose it's down to the user doing it manually, in this case reverting to VFW?

@JN- It's Vegas that has to rely on the flag (together with the user), the codecs know nothing about those flags. The codec gets 0-255 RGB values from Vegas, and knows nothing about input source footage ranges. The codec also doesn't know (cannot know) that of that 0-255 RGB range what is actually valid. 0-255, 16-255, 16-235?

VFW and the plugin works exactly the same way, the ONLY difference is that the plugin doesn't currently have the option for full range RGB-YUV conversion.

In this particular case, VP18 is what actually clips the range of the source media in 8-bit full range mode when the input is set to Limited (the default), since it range expands in that case (so 16-255 becomes 0-278 which gets clipped to 0-255). VP17 doesn't do this, since the engine mode is assumed limited range (8-bit video levels), and all YUV input is loaded as-is without range expansion (so end up in Vegas as 16-255), so for VP17 it gets into Vegas unmodified. I'll elaborate further on this later...

@MagicYUV Ok.

" VP17 doesn't do this, since the engine mode is assumed limited range (8-bit video levels), and all YUV input is loaded as-is without range expansion (so end up in Vegas as 16-255), so for VP17 it gets into Vegas unmodified."

Trouble is that it's still a poorer result in VP 17 plugin than in VFW.

@JN- OK, this'll be long, so here we go.

I'll list the steps happening for both VP17 and VP18 in sequence, indicating at each step the data range and what happens. The VP17 case is essentially equivalent to setting VP18 to Legacy 8-bit mode and leaving the source Color range as Limited.

[A] VP17:

1. INPUT: Source.mp4 [16-255] actual range, flagged as Limited
2. Vegas converts to RGB without range expansion --> RGB [16-255] actual range
3. MagicYUV receives RGB [16-255] from Vegas, then either
   1. VFW codec Full range setting OFF (same as if using plugin):
      1. MagicYUV converts internally to YUV Limited --> YUV [30-235] actual range
      2. OUTPUT: MagicYUV 420 YUV [30-235] actual range, flagged as Limited
   2. VFW codec Full range setting ON:
      1. MagicYUV converts internally to YUV Full --> YUV [16-255] actual range
      2. OUTPUT: MagicYUV 420 YUV [16-255] actual range, flagged as Full

[B] VP18 in 8-bit Full range engine mode, Source Color Range set as Limited (the default in this case):

1. INPUT: Source.mp4 [16-255] actual range, flagged as Limited, treated by Vegas as Limited
2. Vegas converts to RGB with range expansion --> RGB [0-278] actual range, Vegas clips to [0-255]
3. MagicYUV receives clipped RGB [0-255] from Vegas, then either:
   1. VFW codec Full range setting OFF (same as if using plugin):
      1. MagicYUV converts internally to YUV Limited --> YUV [16-235] actual range
      2. OUTPUT: MagicYUV 420 YUV [16-235] actual range, flagged as Limited
   2. VFW codec Full range setting ON:
      1. MagicYUV converts internally to YUV Full --> YUV [0-255] actual range
      2. OUTPUT: MagicYUV 420 YUV [0-255] actual range, flagged as Full

[C] VP18 in 8-bit Full range engine mode, Source Color Range set as Full (NOT the default in this case):

1. INPUT: Source.mp4 [16-255] actual range, flagged as Limited, BUT treated by Vegas as Full
2. Vegas converts to RGB without range expansion --> RGB [16-255] actual range
3. MagicYUV receives RGB [16-255] from Vegas, then either:
   1. VFW codec Full range setting OFF (same as if using plugin):
      1. MagicYUV converts internally to YUV Limited --> YUV [30-235] actual range
      2. OUTPUT: MagicYUV 420 YUV [30-235] actual range, flagged as Limited
   2. VFW codec Full range setting ON:
      1. MagicYUV converts internally to YUV Full --> YUV [16-255] actual range
      2. OUTPUT: MagicYUV 420 YUV [16-255] actual range, flagged as Full

As you can see also, that setting the Source Color Range to Full for VP18 in 8-bit Full range engine mode is exactly the same as if using VP17.

Now that we cleared up what actually ends up in the files, the next important thing to clarify is how the ffmpeg metric operates. The ffmpeg libvmaf and psnr filters only operate on raw YUV values. They don't take any range flag into consideration, just blindly compare the numeric YUV values, regardless of their meaning. So naturally the "best" result was when the YUV data in the MagicYUV output was [16-255] (the same as the Source.mp4) regardless of whether it was flagged as Limited or Full.

To prove this, we can run the filter from the command line. So first, I did a render using VP18 in 8-bit Full range engine mode, with source.mp4 color range set to Full and render through VFW to MagicYUV with Full range setting enabled (This is the [C] case above with the final OUTPUT at the [C].3.2.2. point). The command is:

ffmpeg  -i "vp18-full_src-full_vfw-full.avi" -i Source.mp4 -lavfi "[0:v]null[main];[1:v]null[ref];[main][ref]libvmaf=model_path=vmaf_v0.6.1.pkl:pool=harmonic_mean:eof_action=endall"  -f null -
...
[libvmaf @ 000002950435eec0] VMAF score: 99.750632

Which is precisely what you got in your first screenshot.

However, if we ask ffmpeg to first convert BOTH files to RGB, we get something very different:

ffmpeg  -i "vp18-full_src-full_vfw-full.avi" -i Source.mp4 -lavfi "[0:v]format=rgb24[main];[1:v]format=rgb24[ref];[main][ref]libvmaf=model_path=vmaf_v0.6.1.pkl:pool=harmonic_mean:eof_action=endall"  -f null -
...
[libvmaf @ 000001fbef1c2a00] VMAF score: 73.067005

Our beautiful result is destroyed. Why? Because when converting to RGB, ffmpeg does consider the flags, and since MagicYUV is flagged as Full and Source.mp4 is flagged Limited, they end up totally different. (BTW this is also precisely what happens to the avi you gave me "03 MagicYUV, (8 bit 420).avi" if compared this way).

What's important to understand is that comparing raw YUV values is meaningless without taking into consideration what they mean. In the end, they will get converted to RGB for display, and that is what ultimately matters.

Taking yet another example, if we run the metric blindly again using the VP17 plugin output (the [A] case first OUTPUT above), we get:

ffmpeg  -i "vp17_plugin.avi" -i Source.mp4 -lavfi "[0:v]null[main];[1:v]null[ref];[main][ref]libvmaf=model_path=vmaf_v0.6.1.pkl:pool=harmonic_mean:eof_action=endall"  -f null -
...
[libvmaf @ 000002c0e5898380] VMAF score: 71.349577

A terrible result, but if we first tell ffmpeg that Source.mp4 is actually Full range and then convert both to RGB and then do the metric, we get:

ffmpeg  -i "vp17_plugin.avi" -i Source.mp4 -lavfi "[0:v]format=rgb24[main];[1:v]setrange=full,format=rgb24[ref];[main][ref]libvmaf=model_path=vmaf_v0.6.1.pkl:pool=harmonic_mean:eof_action=endall"  -f null -
...
[libvmaf @ 0000012f7e796cc0] VMAF score: 99.648024

A wonderful score. Now obviously the VFW Full range one is a tiny bit better, since the plugin always converts to Limited, but it's still what we expect, and what it should be.

In summary what's very important to understand here is that running the ffmpeg metric without knowing precisely what is being compared to what is meaningless, because the metric simply compares raw YUV numbers, but those numbers mean very different things on how the file is flagged OR how the files were produced (what inputs were used and how were they handed over to the codec, etc.). So I'd be very careful with using the ffmpeg metric tool, and make sure that all the files being compared contain the same range of values or first convert them both to RGB with the ffmpeg filter chain and see the results that way to make sure I'm not accidentally comparing apples to oranges.

EDIT: Another important takeaway is paying close attention to the Vegas engine mode and the source color flags and how they are imported. If for instance the source is flagged as Limited but in actuality is Full, then all bets are off. The reason it came out looking like as if it was good was because the VFW codec full range setting counter-balanced the incorrect import, but that was just sheer luck, and combined with the fact that ffmpeg metric compares raw numbers without meaning, these 3 mistakes combined gave the impression that it's the best result.

I hope this clears it up a bit. If anything is unclear, just ask.

@MagicYUV Excellent. So looks like i’m back to robbing orchards again. Seriously, if the limited flag was modified in the source to full would that resolve some or all of these issues. If so, how to do? Is there an ffmpeg syntax available to do so without reencoding the source?

"Is there an ffmpeg syntax available to do so without reencoding the source?"

UPDATE: I overlooked that in VP18 the "Flag" can be changed. While in VP Properties, Pixel Format set to full range, then you can change the value to either undefined, Limited or FULL by one of two methods. Select the Properties of the media in the Source Media Window or select the properties from the Timeline by right clicking the video event. This "Flag" change does give a better result, see the #5 example on page 2 of this thread.

@JN- Regarding the Limited flag in the source btw, it is also possible that it is in fact correct! So of the [16-255] range only [16-235] is "valid" and anything above 235 is overshoot. But I don't have enough experience on this, maybe someone else can help me out here?

In any case, if the Limited flag was in fact correct, then everything changes again. Because in this case, the MagicYUV VFW export with the full range flag is definitely wrong, even though the ffmpeg metric comparison says it's right (because again, even though the raw YUV value numbers match nicely, they mean different things compared to Source.mp4).

To be honest, I have the feeling that this is the more likely scenario. I'd be surprised if the mp4 had the flag set incorrectly, that would mean that every single media player simply playing the mp4 would display wrong colors. Which is unlikely.

In general, what you have to ask yourself when doing stuff in Vegas is:

• What is the black/white point (ie. range) of my input? For Limited, it is: 16,235, for Full it is: 0,255
  • MagicYUV 8-bit is ALWAYS considered Full range when importing, since it always gives RGB to Vegas.
  • If a source is flagged as Limited, values above 235 might still appear, but those are above whites.
• What is the black/white point (ie. range) of the current Vegas engine mode? For 8-bit legacy it is 16,235, for 8-bit full it is 0,255
  • Remember, than when the engine is in 8-bit full mode, but a source flagged as limited is imported, Vegas automatically range-expands, BUT only YUV inputs. MagicYUV is unaffected here, because MagicYUV 8-bit always communicates RGB with Vegas, and those are never touched by Vegas automatically.
• What is the expected black/white point (ie. range) of the codec I'll be rendering as? For MagicYUV 8-bit (both VFW, and plugin, for ALL 8-bit variants), it is always 0,255
  • With this in mind, the levels might have to be adjusted, so the black/white points are at 0,255, as the codec expects them

There's still some good info in this article, even though it's 9 versions old, a lot of it still applies today for legacy 8-bit mode: http://www.glennchan.info/articles/vegas/v8color/vegas-9-levels.htm

(One more bottom line to be safe is: never turn on the full range setting in the MagicYUV VFW export options. It should never be required, except for very-very special circumstances. If things magically appear to get fixed with this option, then the problem is elsewhere, most likely somewhere in the incorrect assumptions/settings in the above 3 points.)

EDIT: Well, I might retract this last recommendation, I don't like giving rule-of-thumb recommendations. In most cases, setting Full range is fine, and should work nicely in Vegas and most editing software, since MagicYUV always communicates RGB with editing software, so they never know what's underneath. But when you begin using external YUV comparison tools (like ffmpeg) or more bare-bones apps (like VirtualDub2 or Avyinth and such), then pay special attention.

@JN- Regarding ffmpeg, one easy fix would be to extend the comparison script to yell if the reported YUV ranges don't match. That should at least prevent misleading results.

As an additional remark:

One benefit of using 8-bit legacy engine mode btw in this case is that you get access to the superwhites in the source, ie. the values above 235 (without the need to go to 32-bit floating point mode). In 8-bit full engine mode, they are gone for good right after import, since Vegas range expands and clips them away. But as explained previously, staying in legacy 8-bit engine mode implies 16/235 black/white points, so when doing the ultimate final Render, a levels filter is needed to bring the range to 0-255 for MagicYUV (other codecs might expect different things here).

As you might have concluded from all the above, MagicYUV in it's current incarnation (either VFW or the plugin) is incapable of correctly retaining superwhites, ie. incapable of exporting from Vegas a Limited YUV file with values outside 16-235, because of the fact that it always communicates 0-255 RGB with Vegas, which gets mapped to 16-235.

If that's a reasonable addition to pursue, then maybe the plugin should be extended with additional options and the communication with Vegas changed to allow this scenario. But I'm afraid it has the potential for even more confusion...

@MagicYUV For sure its possible to keep going down the rabbit hole. I just checked again the 10 pieces of original media that I used to create the source files. About 6 of the 10 are 0-255, the rest are about 16-255. When I rendered out the uhd source.mxf it followed very closely the original pieces for levels. The FHD source.mp4 that I use is similar, but a little less, i.e. not quite 0-255, starting greaater than 0 but less than 16.

No attempt was made to conform the output to say limited, although that flag is in both files, all rendered from vp. My primary concern was to have various samples, foliage, water, people, movement etc that I thought would tax any attempt to reproduce accurately, basically make it hard to mimic, make a good copy. These samples I thought were typical of some everyday scenes. Having said all of that maybe it was a bit overkill.

For sure conforming the two source files to 16-235 would have resolved these anomalies with your codec,

“As you might have concluded from all the above, MagicYUV in it's current incarnation (either VFW or the plugin) is incapable of correctly retaining superwhites, ie. incapable of exporting from Vegas a Limited YUV file with values outside 16-235, because of the fact that it always communicates 0-255 RGB with Vegas, which gets mapped to 16-235.

If that's a reasonable addition to pursue, then maybe the plugin should be extended with additional options and the communication with Vegas changed to allow this scenario. But I'm afraid it has the potential for even more confusion”

Decisions decisions...

@JN- Conforming to 16-235 wouldn't have changed much IMO, since if the flag is correct in Source.mp4 (ie. Limited), then it is already "conformed" to 16-235, the fact that it has values outside 16-235 (ie. 16-255) simply indicates that the camera didn't clip at 235, but instead just wrote the overexposed values there. They don't matter though, as they will be clipped anyway sooner or later (the latest when it actually gets displayed somewhere), but could be useful while editing, as you have more dynamic range to work with.

Though I'm not sure where Source.mp4 comes from, but I got the impression that it's directly from the camera, is it not? If not, then again it all depends how it was encoded, whether the Limited flag can be trusted or not...

“Though I'm not sure where Source.mp4 comes from”

@MagicYUV Source.mxf, UHD, was rendered out in vp from 10 sooc (several different cameras, with different levels output) media, to make up a total of 27 seconds, and its near identical to the original media in levels.

Source.mp4, FHD, I cannot remember, maybe sourced from source.mxf, levels not quite as wide as the 10 source media.

VP rendered out both 27 second source and subsequent various 27 second codec outputs.

@MagicYUV I see there is an option for a discount code at checkout for your updated codec. Would you happen to know of anything that may work in there.....? :P

FWIW ... I get these best UHD results using VP18, with the Project properties set to legacy 8 bit video levels and using the MagicYUV Plugin set to 10 Bit YUV 422. Source is source.mxf 10 bit 422.

I tried lots of other combinations including VFW, but the Plugin gave the best results, for this media, and was very fast rendering also on my PC, VFW was slower.

@JN- What is the point of exporting 10 bit when the engine mode is set to 8-bit?

@MagicYUV As I said previously, I did try very many combinations of things, re: Magicyuv, both using several different render options in vfw and also in the plugin, in an attempt to get the best quality result. One of those tests included setting to 32 bit in the vp project properties, but it wasn’t as good as the 8 bit result.

I have two sets of source files, fhd and uhd. Worth to mention that I also use two different tools to test the render quality, ffmpeg and the HO RQ tool, courtesy of wwaag. The latter is a more comprehensive test, and I sort my tables based on the RQ SSIM results.

It takes a very long time to complete a uhd HORQ test, ~ an hour. I just got the results from Gen1 uhd using the best settings mentioned above. Gen 3 now below.

Magicyuv.. MSE 0.069 ..... PSNR 59.786 ..... SSIM 0.9998 Generation 1

Magicyuv.. MSE 0.099 ..... PSNR 58.226 ..... SSIM 0.9996 Generation 3

by way of comparison my results for Sony YUV 10 bit lossless, vfw, were ...

Sony ........ MSE 0.700 ..... PSNR 59.718 ..... SSIM 0.9998 Gen 1

I haven’t yet updated my tables with these results.

@MagicYUV So I made up a new Source.mp4 FHD, also 27s duration. It's conformed to 16-235 and flagged as such in the metadata.

I rendered out 3 files, one file using the Plugin and two files using VFW, both with and without the "full range" box in VFW MagicYUV config box checked.

Using both ffmpeg RQ metrics and HO RQ metrics the best result was using VFW with the config box checked for full range.

The HO RQ results for the 3 files are closer (2 the same) but nevertheless using the VFW still gives the best result.

@MagicYUV

Is there a reason why field order information is not or cannot be added to the metadata for interlaced footage? Since Vegas import always relies on a file's metadata, interlaced footage is always seen as "progressive" and must be changed manually by the user. I see the same thing for other codecs. Just curious as to why this metadata is not available. Thanks.

@JN- VP17 or VP18 and what engine mode was used to render the MagicYUV files?

EDIT: Sorry, I mismentioned jean-Fougas, sorry about that.

@wwaag The plugin should communicate that information, if it's available in the encoded data. VFW has no way of communicating this information.

@JN- VP17 or VP18 and what engine mode was used to render the MagicYUV files?

@MagicYUV I used VP17, keeps it simple, fewer choices. I have no idea what u mean by engine mode?

I used The MagicYuv plugin, plus through vfw. If you mean the render template? then I rendered using MagicYuv 8 bit YUV 420. The FHD source is the same, 8 bit 420.

@JN- By engine mode I meant the "Pixel Format" under "File -> Properties...". VP17 means 8-bit legacy.

Just a quick question, how would you explain that in #1 vs #2 the HO RQ PSNR relative difference is tiny (46.254 vs 47.238), while for the ffmpeg it is huge (32.281334 vs 56.06059)? Doesn't that discrepancy bother you, or raise red flags that something is off?