I'm trying to render a video in xvyCC format and I'm having trouble.

According to the instructions, you have to set "32-bit floating point (video levels)" and that's what I do. I import the mts file from the camera and since it has a Y range of 16-255, I turn on the Brightness filter and lower the luminance. The luminance is within the range, and the chrominance sometimes exceeds the range. That's how it should be.

The problem starts when I want to load an AVI file (encoded UT Video). It's the same mts file, only with improved quality and adjusted Y to 16-235 range. As you can see on its histogram, the luminance is in the limited range.

However, after importing to Vegas on waveform I see that it is way out of range.
And here's the question -- why is this happening and what to do about it?