Converting M2T to DV Widescreen

trynot4563 wrote on 5/9/2010, 12:01 PM
I'm using Gearshift to convert XH-A1 M2T files to SD DV Widescreen AVIs. My final product will be on DVD and will probably require 20-25 tapes of raw footage, so I figured why not save hard disk space and just edit with the 720x480 proxies and bypass the whole HD thing. Seemed like a good idea. Now I'm noticing noise and blockiness on certain takes due to the conversion. Usually on ones that have trees blowing in the background. Most of them look fine, but a few bad apples... Here are my observations/options so far:

Before I go any farther, I'm using Vegas 7.0e, so some of my problems may be solved with an upgrade. Too many M2T's seem to blow it up. But I've never had a problem working with DV files and I hate upgrading unless I absolutely have to and I don't want new problems if I can avoid it. I'm sure you understand what I mean.

So back to my main problem. I thought, Ok, no problem, since there is no way to adjust the DV Widescreen codec I'll use a different codec to create the SD files. I've tried Lagarith, Huffyuv, and Cineform and they all look good, but each one will create a file with a PAR of 1.0 even though I tell the renderer to use 1.212, so all the files come back in scrunched horizontally and I'd have to go through each media instance (or a script more likely) to change the PAR to 1.212. And that would only be a project-specific change. I'd have to do it again in any other projects I inserted these files into. If I used this method, Cineform had the best quality-to-size ratio. If I could only find a way to change the PAR at the file level. I did a quick search on the Web about this, but the answers were beyond my technical pay grade. Lagarith and Huffyuv made such large files that I was better off just creating 720p Cineform intermediate AVIs and working with those (I still might) then porting out to DVD from there.

My next option was to do a "shift gears" with Gearshift back to the M2t file from the proxy, but I'm finding this not reliable because when the project gets either too complicated or too many files, it crashes. I haven't studied this too deeply to know the breaking point, but it's usually not a large project. That may be a Vegas 7 thing, too, since I do have problems trying to do too much with M2T files. I've also tried this on two different computers with the same results, so I don't think it's hardware. I'm a little bummed that the shift gears portion of this program isn't working for me, but the proxy and intermediate file creation has worked fine.

Any ideas?

Comments

John_Cline wrote on 5/9/2010, 12:16 PM
The header of AVI files does not contain a field where the PAR of the video can be specified. The exception to this is a widescreen DV-format AVI file.

Since you can't store the widescreen flag in an AVI file and it's a pain to have to set the PAR manually, why don't you convert them to intermediate files with a PAR of 1.0 to begin with? In PAL, that would be an image size of 1024x576 (or 854x480 in NTSC.)
trynot4563 wrote on 5/9/2010, 1:16 PM
I have tried that. It doesn't work with the Cineform HD 2.5 codec, says the codec doesn't support the render settings. Lagarith and Huffyuv work but the file is almost as big as a 1080p HDV intermediate. I'd go 720p in that case.

Weird (well, maybe not) that the AVI header does not contain the PAR field yet Vegas puts the option in the render dialog.

Thanks.
NickHope wrote on 5/9/2010, 10:19 PM
Those m2t files should convert beautifully to DV widescreen files using the regular Sony DV codec. Make sure you have a deinterlace method set in your project properties ("blend fields" or "interpolate"). If you set it to "none" then the resize is done wrong.

As for your m2t files crashing Vegas, this sounds like the same problem I used to get. There's a limit on how many program-stream mpeg files can be put on the timeline. The number was improved in 8.0c (it's there in the release notes) but there was still a limit. I don't know if this was further improved in 9.0. Depending on how your HDV footage is captured/edited/rendered, the m2t files can either get handled in Vegas by the Sony HDV codec or the older MainConcept mpeg codec. To find out which one is being used, go into the VEGAS explorer window (not the Windows explorer window) and right click > properties and scroll down to the plug-in. m2tsplug.dll is the Sony codec and mcplug.dll is the MainConcept one that has the limit. Capturing with HDVSplit, or rendering in Vegas 8, my files are handled by m2tsplug.dll, but if I run a file through MPEG2REPAIR or VideoRedo or Womble they are subsequently handled by mcplug.dll

The Vegas HDV codec was improved at 8.0c. They improved stuff including the repeat frames at the end of the clips. But to get it I suspect you would have to upgrade all the way to 9.0, which would be something of a gamble for you as SCS still haven't got 9.0 right. If you could just go as far as 8.0 then I would recommend editing native HDV on the timeline which works fine even on modest computers. That's what I do even if DVD is my final product. It's not costing you disk space as HDV and DV are similar file sizes, and HDV is WAY smaller than those other options you're looking at.

By they way, if you do use another codec, I would use PAR 1.1852, not 1.2121. Search those numbers in the forum to see why.
trynot4563 wrote on 5/10/2010, 4:39 AM
Most of the converted DV files do look fine, and I've never had a problem capturing straight DV from my GL-2, but that might be a different thing altogether. Interestingly, speaking of codecs, the codec for the DV widescreen that gives me problems says "aviplug.dll" and the Main Concept one that works says "aviplug.dll" both version 1.2. How does that make sense? How do I know if I'm using the Sony DV codec? I've tried changing the "Use Microsoft DV Codec" option (it's normally off) but have seen no difference. I have deinterlace method set to "Blend."
I would post some pictures of the frames, but I don't know how to do that right now.

My m2t codec is mcplug.dll, but I also use HDVSplit. I remember it requiring me to install ffdshow, but I don't understand how that works at all. Maybe there's a problem there (and that problem is me!)

I have also seen some notes on problems with 9.0, so you see my reservations about upgrading. Don't know if I can find 8, haven't gone down that road yet. I was able to make an easy script to change all of the PARs to 1.212 in the project (I'll read about 1.1852 here in a minute), so I may just use the Main Concept codec and run the script, I'm just wondering what other issues I'll gain from that.

Trying to nail down this HDV workflow has been interesting to say the least. Thanks for all your help.
NickHope wrote on 5/10/2010, 8:31 PM
They're both reporting "aviplug.dll" because that's the codec that Vegas is using to DEcode the files. It's using the same decoder (reader) regardless of how the file was ENcoded.

What do you mean by "the Main Concept one that works"? Exactly what codec is that and how did you render with it?

If you're using one of the DV codecs in the Video for Windows templates in Vegas then it should be the Sony DV codec that is being used (as long as the Microsoft one is disabled in your preferences), but I'm sorry I don't know how to confirm that.

Don't use the Microsoft DV codec. The Sony one is the best there is and was a reason for many people adopting Vegas in the early days. It holds up well after many re-renderings. I'm not surprised you don't see a difference between the codecs after one generation though.

To post pictures of frames put them somewhere like photobucket.com or imageshack.us and then embed using the markup in the sticky thread at the top of the forum.

You would have had to install ffdshow in order to get preview with HDVSplit. I never use the preview because I had corrupted files/dropped frames ("lost packets") when using the preview. Just leave it off and watch the footage on your camera screen.

Interesting that you're using HDVSplit but the files are still getting decoded by mcplug.dll. I seem to remember John Meyer was seeing that too. There must be something about your files that Vegas sees as not HDV-compliant enough to use m2tsplug.dll. With 8.0c or later I suppose they might get decoded by the improved m2tsplug.dll. What format are they? 1080? 720? Progressive or interlaced?

Maybe you could try capturing with Vegas instead and see if your files get decoded by m2tsplug.dll. Vegas HDV capture is poor though. You're going to get messy ends to your files but those can be trimmed as a batch by a script (I have one if you need it. Send me an email via the forum).

Back to the initial point, I still suspect there is something slightly amiss with your render settings or project settings and wouldn't be surprised if it's related to interlacing and field order.
trynot4563 wrote on 5/12/2010, 1:28 PM
Sorry, I didn't elaborate on the Main Concept codec. I'm actually using the Main Concept HD 2.5 codec and just changing the size to 720x480 23.976fps progressive. Works great except for the widescreen flag problem mentioned earlier. The size is about 20% larger. My source is a 1080p (Canon XH-A1 HDV 24F) m2t file. My project settings are progressive and blended and I render progressive. And I'm not using the Microsoft DV codec

Here is a picture of a bad frame using the NTSC DV Widescreen AVI template:


Again, this is sporadic. Most clips look fine, but when the problem occurs, it's throughout the whole clip. This happens rendering from the m2t and an HD intermediate AVI.

Here is the same frame (maybe one frame off) using the Main Concept HD 2.5 codec at 720x480. Much nicer:


I remember that about ffdshow now, that it's just for the display in HDVSplit. I wasn't using the display anyway, so I uninstalled ffdshow. I captured some clips using Vegas, same thing, says mcplug.dll. Now remember this is Vegas 7.0e, I don't know if the m2tsplug.dll was up to speed with that release even though it exists on the harddrive. Of course, this is unrelated to my initial problem.

The one thing I haven't tried is rendering this clip on my other system to see if it occurs there. I'm thinking with the trees and movement that there is a lot going on here and the codec can't handle it because this seems to occur on visually complex shots.

Anyway thanks for the help.
NickHope wrote on 5/12/2010, 10:48 PM
Well yeah, that's bad blockiness and quite surprising. I wonder if other forum members have seen such a bad result of a 1080p to DV conversion.

I'm still not sure which Main Concept codec you mean. Is it an AVI codec? MJPEG? Bundled with Vegas? Well, if it works and the file size is only 20% larger then that seems like a workable solution.

Personally I prefer to work in native hdv m2t on the timeline and downsample at render time (I frameserve to CCE Basic to make my MPEG2 files for DVD). I would say if SCS fix the slow project load times and finally get a solid 9.0 release then you could upgrade and use a similar workflow.

I'd be happy to try a 1080p > widescreen DV conversion in 8.0c if you could get a small sample file to me. I could also tell you which codec it's using to read your HDV. Or why don't you check out the trial version of 9.0 which is fully functional?

One final thought. Does your cam let you output DV instead of HDV? In other words an in-camera downscale. My Sony Z1 does and when I was using Vegas 7.0 on my old laptop I used to use those DV files for making DVDs and then also as proxies for editing before archiving HDV stock. The camera's conversion was good.
trynot4563 wrote on 5/13/2010, 4:02 AM
The Main Concept codec came with Vegas 7.0. It's used to make the HDV intermediate files in AVI since the M2T files are hard to edit and, in my case, crash Vegas at a certain point. It's version 2.5 which is actually old. I demoed a version of Neoscene and it upgraded the codec to a higer number. Unfortunately, it didn't do it for me (I think the audio wasn't syncing, I can't remember), so I unloaded it. But it didn't convert my MC codec back and I had to reinstall Vegas.

I'd love to work with the native files, but at this point it's not an option for me. That's where Gearshift came in, but again, converting from proxies back to M2T's isn't so reliable. I mostly do small projects that allow me to either use HDV intermediate files (which are quite huge) or convert back to M2T's and then render out to DVD or an HD format if shown from a computer.

I'll make a small sample of an offending clip and send you a link if you're curious.

As far as in camera down-converting, a couple of things there. The image was really soft when I tried that with another project and it doesn't keep the 24F frame rate. I thought Vegas would reassemble the frame rate from the 30p, but if it does, I can't figure it out. I could do it in TMPGenc, but what a hassle.
trynot4563 wrote on 5/13/2010, 4:25 AM
Ok. Something interesting happened. Rendering a smaller clip in Vegas from the original m2t file used the m2tsplug.dll plugin. It didn't do anything for conversion to DV, but it might for the Gearshift problem if I can figure out a way to easily use that codec. It seems like I keep adding steps to my workflow. Eventually I'll get it down. I THINK this HD camera was a good idea, lol.

I'm uploading the clip now. It should be there at 7:30 EDT 5/13.
http://trynotproductions.com/video/hddance.m2t

I'll try the Vegas 9 trial when I get a chance and see if that helps.

Thanks.
trynot4563 wrote on 5/13/2010, 5:10 AM
I also just realized that it would help if I sent a clip directly from the camera:
http://trynotproductions.com/video/X10011-0133.m2t

It's about 13Mb, the smallest one I could find.
musicvid10 wrote on 5/13/2010, 9:05 AM
Your camera clip is 23.976.
OGUL wrote on 5/13/2010, 10:15 AM
With my HC9E, I can capture either m2t or dv.avi!
Are there any points that I should make attention??
NickHope wrote on 5/13/2010, 10:15 PM
Ah, you mean the Cineform HD codec, not Main Concept.

In Vegas 8.0c, both those clips read as HDV 1440x1080 23.976p. They are decoded by m2tsplug.dll. The first clip has 2 repeat frames a couple of frames before the end.

I put them on my timeline, matched my project properties to them, and rendered them with the "NTSC DV widescreen 24p (2-3 pulldown)" template. I put the rendered files in a new project and again matched my project properties to them. They are perhaps a little soft (e.g. the dancer's face in the camera clip) and there is slight blockiness on one or two frames but not as bad as your frame grab above. Very possibly acceptable, depending on your project.

Is that the template/workflow you used?

I also rendere to Cineform 480p (my installed version is 3.3) and, once the AR is changed to widescreen, I agree, it does look slightly better than the DV.

I would steer clear of 720p intermediates. A two-stage downsample from 1080 to 480 isn't going to help.

Have you found a pattern in which of the Sony codecs handles your m2t files?
trynot4563 wrote on 5/14/2010, 4:38 AM
You are absolutely right. I don't know why I couldn't get the "Main Concept" phrase out of my head. It must be talking about M2T and DV problems in the same breath, lol.

Seeing that both clips are decoded by m2tsplug.dll in 8.0c shows that something has improved (or changed) with the newer Vegas version.

Your test is basically the workflow I use (did use). Again, I'll probably use the Cineform (not Main Concept) codec for now until I find that the new 9.0e upgrade is suitable. And that's good to know about using 720p intermediates. There's too much to learn.

The only pattern I have found with the Sony codecs in 7.0e are that from the camera Vegas uses mcplug.dll and from a rendered clip to m2t it uses m2tsplug.dll.

I think we can put this one to rest. I actually hope that I can edit with m2t's directly once I upgrade and if not, that I can use Gearshift without errors. I did see a script called "Proxy Stream" on this forum. I'll have to look at that, too.

Thanks again for all your help.
trynot4563 wrote on 5/15/2010, 6:16 AM
Installed 9.0e trial. Quick test with m2t to dv conversion: still looked the same as 7e, yucky. Rendering a file with Cineform HD codec still reported as 2.5 version, same as in 7e. Is this correct? A little confusing that there is not even a minuscule version change of this codec from 7 to 9. Decode plug-in for avi is still 1.2 as in 7e but has a different build number and m2t plug-in is now compoundplug.dll.

When rendering with Cineform HD codec what version do you have?

Sorry for the quick short hand note, more later.
NickHope wrote on 5/15/2010, 6:49 AM
It used to be possible to update the Cineform codec by installing the free Cineform Neo Player software but at some point they disabled encoding.

I've just emailed you the version 3.3 codec which came with one of the versions of Neo Player before the encoding was disabled. I sent it to the email address on your Trynot Productions website. You should be able to just swap it out but you will need to rename it. I don't know if the encoding is significantly improved or not.
trynot4563 wrote on 5/15/2010, 8:57 AM
Thank you. I'll give it a try. It's not that the 2.5 version doesn't work well, I was just surprised that installing 9.0e didn't upgrade it. Now I see at some point it did upgrade to 2.8 (in 8.0?) from your message, so I wonder why it didn't install? I haven't found anything about it yet in the forum upon a cursory glance. Do you know of a list that shows what codecs and versions are supplied with Vegas? Either I need to uninstall 7 and reinstall 9 (don't want to do that option since I'm just trying 9 for now), or the trial version doesn't install everything (not sure), or something is just screwed up somewhere with my installation (very possible). The mystery continues.
NickHope wrote on 5/15/2010, 1:02 PM
2.8 may well have been either a typo in that old email I forwarded to you, or it was an upgrade that I got by installing Neo Player at some stage and I assumed it was what had been shipped with Vegas. I used to constantly update Neo Player as Cineform updated it so that I'd have the latest codec (until the point they disabled encoding with it).

I expect Sony did a deal with Cineform for the shipped 2.5 codec and they never updated it. It's not really a suprise. Many of the codecs in Vegas don't get updated across numerous versions of the software.
trynot4563 wrote on 5/15/2010, 1:17 PM
It seems that a 9.0e upgrade may solve my workflow issues. Editing M2Ts is greatly improved and hasn't crashed yet. If I can just work with those, I won't need the proxies (still blocky on the DV Wide 24p AVIs) or intermediates. And I'll have the ability to port out to SD or HD without much heart ache (or heartburn).

I've started a new post about file logging that has always been an issue for me.