Things get curiouser and curiouser.
If I capture the color bars generated by my HD110 in HDV mode onto an m2t file, then drop the m2t file on the timeline and look at the Waveform monitor, I see that the waveform extends from RGB16-RGB235. Furthermore, the color bars don't line up with the preset values on the WFM.

Now, if I convert the m2t file with cineform HDlink, and make sure the cineform codec is set to ITU709, then drop the CF intermediate on the timeline, I see that the waveform still extends from RGB16-RGB235, but now, the colorbars line up perfectly with the preset values on the WFM.

In fact, if I load the m2t file into womble MVW, and examine the color bars with an eyedropper tool, it shows the blacks at RGB0 and the whites at RGB255.

What's going on here? It looks like Vegas still doesn't map m2t files to the correct color space. And even Cineform intermediate luma values are incorrectly mapped to ITU 601.

I don't see RGB values displayed in the Sony Waveform. Do you mean the Sony Histogram display scope?

Do the two clips look the same?

Rob Mack

I think you're right by design. Really straining the grey cells here but think I remember this being mentioned in the early days of HDV support. Vegas maps 709 to 601 for internal display and metering.
Keep in mind that you can mix and match DV and HDV on the one T/L. Imagine what would happen if your scopes and monitor was jumping between 709 and 601 all the time!
I think it all comes out OK in the end though. For HD output it all goes out correctly as 709, for DV output then it all goes out as 601.
I presume though if you're monitoring through HD SDI everything there would be in 709.

Rob...

Sony WFM doesn't show RGB values, it shows %IRE. Nevertheless, if you have the VideoScopes setting set to Lab Color (RGB 16-235), then 0%IRE=RGB16 and 100%IRE=235.

Bob...

Yeah, you're probably right about the display being remapped to ITU601. Still, something doesn't seem right if raw m2t file chroma reads differently on the WFM than an intermediate.

OK, as a test, I rendered an m2t file(that showed luma range of RGB16-235 on the WFM)out to DVDA MPEG2 template. I, then imported the results into Womble and examined the luma range with an eyedropper tool. It appears that you're right, Bob. The re-rendered mpeg file retains the luma range of RGB0 -RGB255.

I believe I've read that Vegas slips into the 709 experience when you go above standard def resolutions. Anything above that whether interlaced or not. Not sure what the calculation does with pixel aspect ratio.

It might be a concern if you DI at 960x540 for example.

true or not, it is important that you calibrate for the new gamma. Whether you are on an LCD/plasma monitoring 601 or on a CRT monitoring 709 HD/HDV/AVCHD.

It isn't the integer range in the files that is important, it is the multipliers that give you 'voltage' on the vectors and scaler signals that would result in Y'PbPr or RGB.

Apart from staying legal (for which the headroom when you swap between could be a struggle), do you color correct to formal color palettes based on a table of numbers or do you use your eyes? ;)

It has been a problem for as long as I've been in this business. Adam Witt wrote about it a decade ago. In the days when TV monitors and NTSC broadcast were the only available distribution media, it was important to maintain the RGB values in the 16-235 range(and, I might add, a pedestal of 7.5).

Nowdays, with HD displays and advanced encoding algorithms like MPEG, it's entirely possible to have a display system capable of showing RGB 0-255. All video cameras that I'm aware of record in the full range of RGB.

In digital still image printing, most modern ink jet printers are capable of printing RGB values from 5-250. So, why am I throwing away 15 points on either end of my available dynamic range? Video camera sensors are notoriously low on dynamic range to begin with, without making it worse by remapping values to 16-235. To make matters worse, if I do the transformation incorrectly, I further degrade my image, either pushing it into being washed out or muddy dark.

Admittedly, if my final destination was NTSC broadcast, I'd have to do it anyway.

I always end up comparing my settings to a formal color palette on a calibrated monitor. Having said that, I also always depart from the technically correct formality and adjust to taste. I get MUCH better results. I should add the caveat, for the sake of those who distribute exclusively on NTSC, that my distribution is not necessarily for NTSC applications.

Having said all that, the color mapping differences between DV and HDV are significant. Inappropriate application of the wrong standard, will result in colors being wrong. As Glenn Chan so appropriately pointed out, the calculation of Luma from RGB values, is signifcantly different in the DV algorithms, as compared to the HDV algorithms.

For NTSC broadcast, the end is nigh.

Mere months away (well, beginning of 2009 anyway).

Unless the spineless politicians give in again, but that seems unlikely since there are large profits to be made from the analog frequencies, so our politician's pockets will be well oiled if there is any citizen resistance to the changeover from NTSC to ATSC.

Bill and I have a discussion here:
http://www.dvinfo.net/conf/showthread.php?t=102164

To correct a few things:
--DV and HDV use Y'CbCr color space. The legal range for Y' is ALWAYS 16-235. Black level at 16, white level at 235. Values outside that range handle illegal values / overshoot / undershoot.

709 does not use lab space, and it does not use a 0-255 range. Legal Y' range is from 16-235.

--Y'CbCr values can be mapped to computer RGB (0-255) or studio RGB (16-235). Computer RGB has black level at 0 and white level at 0. Illegal values are clipped off. Studio RGB has black level at 16 and white level at 235... it retains a lot of the important illegal values.
Which mapping is used depends on what codec you are using.

You can make an argument for going to either color space (or a different color space).

--The reason why Y'CbCr devotes values to illegal values is because it may be used in conjunction with analog equipment. The calibration on analog equipment may drift, resulting in levels that are slightly off. When you convert those values to digital, there will be clipping unless your digital format accomodates this by having footroom and headroom.

You can also do things like send color bars... which have a blacker-than-black segment (which is illegal and would otherwise be clipped).

Thanx, Glenn.
First, I must reiterate my apology for using the term "lab color". What I meant was Studio RGB. I spend so much time with Adobe Photoshop that I confused the two and they are not synonomous.

I would, if I may, argue a few small facts, towhit:
1- Computer displays are RGB 0-255. If one wants to use the full dynamic range of computer displays, that should be an option. Yes, clipping will occur on an NTSC display, but, modern Hi-Def displays will handle 0-255.
2-The HDV spec makes provision for RGB 0-255. My understanding of this provision of the spec, at least technically, is that this is 10-bit. Admittedly, I'm not entirely clear on when RGB 0-255 is allowable. Video cameras like the HD110, which is 12 bit, use a full 0-255 RGB color space.
3-Sony Vegas 7, currently ,does NOT handle 'raw" m2t file color space properly. Black is mapped to RGB 16, white is mapped to RGB 235, but the luma is incorrectly calculated for color. This may or may not be due to an incorrect header cue from the camera.
4-Regardless of the source color mapping, Sony Vegas will remap everything to RGB 16-235, by default. No option is provided that allows a 0-255 color mapping. Womble MPEG video wizard, for example, allows RGB 0-255.

caveat: My statements are based on my current understanding.One of the reasons for posting these things, is to invite anyone who knows, more definitively, to please provide input and/or corrections.

2. HDV certainly is NOT 10 bit! The HD110 might use 12bit processing but still records 8bit, some cameras use more than 12 bit processing and still record to 8bits, the two number have absolutely no connection directly.

Really the rest of it is just a mess, sorry.

DV will quite happily record 0 to 255 and Vegas will not raise a sweat over it, if you don't believe me just drop some default Black out of the Generated Media on the T/L and record it to tape, or for a real thrill try a checkerboard pattern, that'll give you superblacks and superwhites. You might give the average TV a case of heartburn of course if you feed it that from a composite connection!

As Glenn pointed out the DV spec was written way back and allowed enough headroom for the analogue circuits, much as USA NTSC has setup but when Japan got around to it they dropped the setup. It exists purely because of issues with old technology and sync detection.

I haven't a clue where you're getting all this Vegas is remapping ideas from either, if it was the generational loss would be hideous and it isn't. Vegas allows RGB 0-255, in fact by default that's how it works and it's one thing one could technically, in some respects complain about.

Where the issue came from was in analogue video land there's voltage levels lower than black and whites can sometimes overshoot or things get out of calibration. For safety, to avoid clipping headroom was allowed. Here is a good reference.

I kind of agree with you about these standards being a smidge wastefull of dynamic range but it's a bit late in the day to be rejigging the whole friggin system, the chaos would be huge. At the same time I know of no system that doesn't have headroom at both ends of the scale. The Cineon standard does the same thing and for other good reasons, processing and grading can push thing outside the limits and you really want to avoid overflows. The alternative is having to remap everything prior to processing and that's I think going to be fraught with even more issues.

Bottom line is, the system works, DV, HDV all goes through Vegas just fine. Sure if you're seriously planning on a film out you may eck a little more out of the image using a fatter pipeline but the costs, complexities and possibilies of screwups increase dramatically. From what I've read trying to grade DV or HDV like it's film just doesn't work, if it did there'd be a lot of very expensive cameras sitting idle. If you're sweating this much over the image you're going to achieve a lot more starting with a better camera system and lenses. Just my opinion here and I know others will disagree strongly, if you're going for a film out then shoot for it and wear the costs of doing it right. Printing to film, even before you get a release print is very expensive. I suspect of all the productions that start out thinking they'll do a film out but aren't financially commited probably less than 1% end up doing it.

Bob.

Thanx, Bob. You've made some very valid points.

Kudos to Glenn and Bob for their excellent expositions in this thread, very helpful!


From what I've read trying to grade DV or HDV like it's film just doesn't work

This is by degree though, not an absolute.

I really take my hat off to Stu Maschwitz for coming up with a simple workflow to grade DV, HDV, etc. in After Effects, somewhat like it was film.

It does produce visibly better quality and is very easy to do, especially if you use the free AE scripts on the CD-ROM in the back of his book, "The DV Rebel."

(The workflow could be modified to work in Combustion or Fusion too, the key is working in high bit RGB. Maybe even V8 will join the club someday...)

Sony Vegas 7, currently ,does NOT handle 'raw" m2t file color space properly.
It does, with the possible exception of JVC cameras.

Black is mapped to RGB 16, white is mapped to RGB 235
This is fine / correct.

, but the luma is incorrectly calculated for color.
Only for JVC cameras.

This may or may not be due to an incorrect header cue from the camera.
It might be the codec's fault in that it is not reading the Rec. 601 luma co-efficients flag correctly. Or something like that. I don't know that much about how HDV should be formatted and I've never read the HDV spec.

4-Regardless of the source color mapping, Sony Vegas will remap everything to RGB 16-235, by default.
No, the behaviour is dependent on the codec being used.

No option is provided that allows a 0-255 color mapping. Womble MPEG video wizard, for example, allows RGB 0-255.
One workflow that works is this:
Shoot HDV. (Assuming it's a camera that uses Rec. 709 luma co-efficients.)
Ingest into Vegas.
Vegas's HDV codec will decode the Y'CbCr values to StudioRGB.
You encode to MPEG2. The main concept codec will take the studio RGB values and convert them to Y'CbCr.
On playback in a DVD player, those Y'CbCr values will get converted to computer RGB (0-255) values.

As measured by Waveform Monitor (no flags set) in Vegas 7d, my Sony Camcorders TRV33 (handycam) and Z1 (prosumer HDV&SD) both record blackest-black (dark room, shutter on, everything stopped down etc.) at 6.2%=16/255 and totally overexposed (everything open, slowest shutteretc.) at 100%=255 (approx.). At normal full (as opposed to over) exposure e.g. when the Z1's zebra stripes just appear, level is 92%=235/255. Neither camera appears to use the 0..16 space and I can't imagine how to record blacker than black. So headroom exists but there are no "feet" to put in the "footroom".

Color bars would have blacker than black.

If you are dubbing from analog to digital, the analog format can have voltages below black level.

"but the luma is incorrectly calculated for color.
Only for JVC cameras."

Glenn, I've got me a JVC camera (HD100) - how is the luma incorrectly calculated for color? Should I be correcting for something? Thanks for any info -

the error is ONLY for m2t files placed directly on the timeline. whether the problem is limited to JVC cameras, only, remains to be seen.

the simplest solution is to convert to an intermediate format with Cineform HDLink.

It would be very difficult to compensate for the "bug" with CC, since the error is non-linear across the color scale.

Bill,
Is it possible for you to post up a short clip of color bars recorded with the JVC camera?

2- I think it's possible to correct the color in Vegas. I just have to crunch the numbers.

The error is not non-linear.

Thanks for the input guys - I will try and do some checking and try and post some JVC color bars, etc (probably tomorrow).

I will say that the footage from the JVC, either as raw m2t or Cineform intermediate, has looked really good. What kind of error behavior should I be looking for?

Is this close enough...? It might be that you can get a little closer if the channel blend filter let you input more precision / decimals. I'm not sure. Or it might be clipping that's the issue. Again, I haven't worked it out.

Add the channel blend filter

Red = 0.914 * (R)
0.078 * (G)
0.008 * (B)

Green= -0.105
1.172
-0.067

Blue = 0.010
0.032
0.959

EDIT: Fixed an error in Green.

Glenn...
i've sent you a private email with the download instructions for the file you asked for. I applied the color corrections you provided above; and, if I did it right, it doesn't come close. This leads me to suspect it's not color space coefficients. I sure hope there isn't something in my own process, after all this, but, I don't see what it might be.