How to do lossless PAL DV .avi to .mov?

NickHope wrote on 4/17/2010, 10:57 PM
A Mac-based stock footage rep needs Quicktime .mov versions of my (several thousand) PAL DV .avi files. Naturally I want to do this losslessly, with just a simple remux from .avi to .mov.

However, Vegas appears to be doing some recompression, even with like-for-like settings. Once the field order is corrected on the timeline, there are still visible differences from the original avi, where I'm assuming lossy recompression has taken place. It's softer, as were files rendered with the Avid QT DV codec when I tried that.

So I tried to do a lossless remux by doing simply "Save as" to .mov in MPEG Streamclip and Quicktime Player. I also did it with FFMpeg by using a straight "-vcodec copy" command line (via the WinFF GUI). This thread on doom9 states that lossless conversion should be possible, but in each case I get a significant and unacceptable increase in brightness. I can see this increase in brightness on the Vegas 8.0c timeline, or in media players including the Quicktime Player (Windows). Blown out highlights, plus an increase in contrast.

But I also see this same increase in brightness when I play the original avi in Quicktime Player. So it seems to me that instead of doing a simple remux, each of those non-Vegas methods I tried is reading the .avi and somehow changing the luminosity before muxing to .mov.

Does anyone know how I can avoid this?

Comments

Opampman wrote on 4/18/2010, 6:56 AM
Nick - if I'm not mistaken, I believe this was a known issue with QT discussed here recently - at least for some version of QT. I'm probably wrong but you might want to search the forums.

Kent
musicvid10 wrote on 4/18/2010, 9:30 AM
Nick,
Can you email me a short sample (<20MB)? You have my address. Same confidentiality arangement as before and I'll permanently delete your clip when done.

I'd like to try a couple of things when I have time on Monday. I'm pretty sure you're not going to get a smart render to MOV in Vegas. I think the discussion Kent is referring to has to do with sRGB->cRGB issues in Quicktime. You might want to check some of Laurence's posts if this seems worth following up.
m
NickHope wrote on 4/19/2010, 10:34 AM
Kent, Thanks but I couldn't find any specific reference to Quicktime and levels on the forum.

Mark, I emailed you a sample file.

I looked at the files in VLC to try and circumvent the Quicktime decoding/display and the "simple remux" versions look the same as the original avi, but my Vegas-encoded version looks darker. It's very confusing now to understand what exactly is happening. Seems to be a combination of QT decoding and display issues.

As this is for stock footage, I guess I should probably just do it the "simple remux" way, i.e. theoretically the least lossy way, and accept that the luminosity on the end users' systems is somewhat out of my control.
PerroneFord wrote on 4/19/2010, 10:39 AM
Nick, send me a message, I'll help you on the workflow. But there is no simple remux process for this, and there is going to be a real transcode in the process. It can be done without loss though and perhaps even without gamma shift.
musicvid10 wrote on 4/19/2010, 4:35 PM
After playing with this for a while, I'll agree with (and defer to) Perrone's experience with this stuff.

It seems QT needs an apple-flavored codec for DV-MOV files, and there is a gamma shift in the transcode; it actually seems more pronounced with the ffmpeg based converters I tried (got brighter).

The closest off-the shelf match I found was using the DV-PAL codec with the MOV template in Vegas. Although a tiny bit subdued in the highlights (in VLC), it kept all the detail. My settings were Best, LFF, DV-PAL, 24 bit.

EDIT: That Avid 1:1x codec looks even a bit closer, and is a 99% match on the scopes. Settings SD-PALi, RGB space, but that is 170Mbs, not DV25.
PerroneFord wrote on 4/19/2010, 5:52 PM
I was going to try the Avid 1:1 but you beat me to it. I generally find that Avid's codecs do a terrific job keeping gamma shifts to a minimum. But the require a real encode. I was also going to try an IMX encode and see if that flies.

Moving anything into quicktime and out is always problematic, and it's why I use DNxHD for HD stuff when I have to do this, because the gamma shift is nil. Avid did all the hard work already to sort this stuff out.
musicvid10 wrote on 4/19/2010, 6:07 PM
Agreed. Using the Avid 1:1x as an intermediate for SD MOV is a new revelation for me! Using the RGB space, it is astounding how close it is visually and on the scopes.
And trying the IMX is a terrific idea too, I would especially watch the near-whites when testing.

That's why I like fooling around with other people's problems so much -- even if I don't stumble across the ideal solution, I always gain a ton of new information and ideas in the process.
PerroneFord wrote on 4/19/2010, 6:57 PM
That's why I tackle a lot of these problems too. And I've learned a TON in the process. I would have NEVER discovered the Avid codecs had I not been working with an FCP guy and trying to find a solution for round-tripping between FCP and Vegas.

The SD codecs installed with DNxHD, and I just decided to try them out one day. Darned impressive.
NickHope wrote on 4/20/2010, 2:06 AM
Thanks very much for looking into this guys. I really appreciate it.

Perrone I sent you a message.

The guys at the other end tried a DV avi to DV mov conversion in MPEG Streamclip on the Mac and the result is the same. Below are Vegas grabs of the original DV avi followed by the converted DV mov clip. You can see the highlights are blown out on the divers' hands etc..


PerroneFord wrote on 4/21/2010, 7:11 AM
Workflow sent via email.
musicvid10 wrote on 4/21/2010, 9:06 AM
Perrone, Would you email me your workflow also? I would like to have it for future reference. My username at ******.
NickHope wrote on 4/24/2010, 2:25 AM
In the settings that appear to be relevant to luminosity, Perrone's settings are much the same as the ones I have been using, except he used 32-bit floating point pixel format with 1.000 (linear) compositing gamma (I used 8-bit). Also he's on VP9.

These are my settings in Vegas Pro 8.0c:



For an all-Vegas workflow this is fine. There is some recompression going on bit it's barely noticable with standard definition, and the .mov is the same luminosity in Vegas as the original .avi.

But I'm still not totally convinced this is the way to go. Looks like the luminosity is going to vary depending on the NLE or player, so there seems to be no right or wrong answer. As I said above, in VLC Player (which uses its own libraries instead of QT decoder) the Vegas-rendered mov file displays darker than the original avi file, whereas the mov file that has been simply remuxed in Quicktime is the same luminosity. It's Quicktime decoding and playback that displays the simple-remux version unacceptably brighter. So presumably FCP would do the same as they're both Apple??? Or maybe on an Apple, the simple remux version looks the same as the original???

Here's a summary of what I'm seeing:



Anyway I'd hate to think of a Vegas user licensing my footage and getting those horrible blown out highlights, so I'll probably just go with the settings above and render in Vegas.

Do you guys think it's worth using 32-bit mode for CC and render of DV and HDV in 8.0c? I failed to see any difference in the result for DV.
musicvid10 wrote on 4/24/2010, 7:41 AM
Nick, the settings you used, and the observations you've made, are all 100% identical to what I came up with.

I was unable to detect any difference visually or using the scopes between and 8-bit and 32-bit project settings. I assume this is because Vegas is rendering to 8-bit -> 8-bit DV25.

On my Windows monitor, the "slight" darkening of the Vegas-rendered DV is preferable to the considerable lightening of the transcode to Apple DV, whether done in QT, Super or another ffmpeg solution. This is especially true since QT Player lightens and flattens the playback (in Windows, anyway). There "may" be a way to control the colorspace/gamma shift in a command line utility, but you would know more about that than me.

Or maybe you don't need to do anything because of the setup gamma differences between Windows (2.2) and Mac (1.8) displays. You probably need to check your results on a properly calibrated Apple display before getting too much deeper, or ruling out the lighter Apple transcodes completely.

As a DV->DV solution, Vegas came the closest in all of my tests. I'm seeing a slight loss of sharpness by this method however (I also used still captures in Photoshop at 300% as a comparison).

As a lossless solution, using Avid 1:1x was perfect visually and on the scopes, stunningly so. I suggest this because lossless delivery will give your client transcode and playback options in a Mac/FCP environment that we are unable to test in a Windows environment. I know that QT player does not behave identically in both environments, for instance. The disadvantage in your case is the 6x larger file sizes using the Avid 1:1x codec.
NickHope wrote on 4/25/2010, 11:32 PM
Thanks Mark. The Avid 1:1x results do indeed look identical to the original. This is excellent to know, and would be a great solution to a one-off conversion to an Apple user, but since a) as you say, the files are nearly 6 times the file size of DV, and b) most customers would need to download an extra codec to decode it, it's not really the right format for sending thousands of clips to a stock footage house.

I did a little further comparison between the Apple's and Avid's Quicktime DV codecs. To my eyes, the Apple looks slightly closer to the original. In addition I'm losing a line at the bottom of the image when I use the Avid codec (it's black). Quick question... is the Avid DV codec required to decode files encoded with it or would the regular QT DV codec handle it?
Rory Cooper wrote on 4/26/2010, 6:33 AM
edit blablaa blaa cut

short and sweet ........http://www.videocopilot.net/blog/2008/06/fix-quicktime-gamma-shift/
musicvid10 wrote on 4/26/2010, 6:55 AM
Nick, without knowing your relationship with your client, I was imagining that you might be able to send them a lossless intermediate to work with, knowing that FCP won't be as likely to mess with the gamma as it would with Windows-centric DV AVI.

But if the stock house is not able to do the conversion on their end, and instead wants delivery-ready material from you, one of the DV-MOV options you discussed is going to be the best all around.

Let me know how you come out on this one. I hope they are paying you a ton for the distro rights and residuals because the clips I have seen on Youtube are outstanding.
m

EDIT: The Avid DV material will (should?) play with any MOV player; however, I agree with you after testing that Vegas' DV-MOV looks a little better on the screen and on the scopes.
NickHope wrote on 4/26/2010, 11:18 AM
Thanks Rory, I tried that on both my DV avi file and the remuxed DV mov file but it didn't correct the gamma at all. So basically it doesn't work. Didn't try it with non-DV material though, and will bear it in mind when I'm back with H.264 gamma woes one day. By the way I'm using QT Pro 7.6.2.

Mark, I guess I could send them Avid 1:1x and have them convert to DV .mov (they would likely use MPEG Streamclip or Compressor on the Mac) but there will still be a transcode required to get the footage into DV .mov and who knows what might happen there. For the time being I'm going with rendering to QT DV-PAL in Vegas, which is acceptable enough. I keep control of the quality and need fewer hard drives to keep and deliver everything.

Representation-wise, I'm happy enough with the deal, and I definitely need help with the marketing, business and delivery side of things. I wanna go diving and not deal with video formats 24/7!!
musicvid10 wrote on 4/26/2010, 5:25 PM
For the time being I'm going with rendering to QT DV-PAL in Vegas, which is acceptable enough. I keep control of the quality and need fewer hard drives to keep and deliver everything.

Agreed.

I definitely need help with the marketing, business and delivery side of things.
Send me an email about your needs in plain talk. I'm just wrapping up a 1/2M project with a nonprofit, and need something new to worry about.