HDV to tape: too many problems for native editing

Laurence wrote on 4/3/2008, 1:32 PM
I have a new rule of thumb regarding HDV. It goes like this:

If you capture HDV from tape, always transcode into Cineform codec AVIs. There are just too many formatting problems to work with the native m2t clips that were aquired from the tape.

If you shoot the HDV tapeless with either a DR-60 or the compact flash unit that comes with the Z7 or s270, go ahead and use the native m2t clips. They will work just fine.

Comments

Spot|DSE wrote on 4/3/2008, 1:38 PM
Laurence, this is a little confusing. What goes to CF card is exactly the same as what goes to tape. Exactly. Bit for bit.
Where they're different, is that tape can have dropouts. If you're having dropout issues, perhaps it's the tape you're using?
farss wrote on 4/3/2008, 1:56 PM
Here's the confussing part. Theory is that HDV is more robust than DV. That theory does hold up, I've seen way more dropouts in DV than HDV. When HDV dropouts become apparent i.e. the error correction can't correct then they're really bad.

It seems the problem lies in correctable errors on tape. Perhaps error correction is not done with firewire capture and it's the error correction that's causing Vegas to crash. This might not be Vegas's fault entirely, it could have something to do with the MC codec. Ppro users seem to be having the same kinds of issues. It'd be an interesting experiment to try using the files that crash Vegas in Ppro.

Bob.
fldave wrote on 4/3/2008, 2:57 PM
I've had my FX1 since they first came out. I have been doing 95% HDV vs 5% DV for 3+ years. Granted, as this isn't my day job, I have spent countless hours editing/working with HDV footage.

I've never had any dropped frames, black frames, formatting issues at all.

I have never purchased the Cineform product(s) that so many people had issues with back in ?early version 7?? I never had any issues that people were reporting.

I have only recently used HDV Split, and that was purely in a monitoring capability.

I've never used Womble either.

I've had no (zero) HDV problems using FX1, HC7, Sony Vegas 5, 6, 7, 8, VirtualDub, AVISynth, Frameserver, HuffYUV.

I have to say that there is a good chance that there is nothing wrong with HDV on tape, that there are other combinations of issues to measure.
Laurence wrote on 4/3/2008, 4:10 PM
I have had complete and utter frustration ever since I entered the world of HDV. This is across several computers and cameras. I have a new Core2Duo laptop computer, SATA drives, a new Z7 camcorder, and I am still having exactly the same issues. I can work around them with quite a complex set of rules. I have read a bunch of posts by people having the same problems as myself and pretty much all the people who are having problems are facing the same parameters of working around them as myself.

It kind of drives me nuts that some people have none of these issues. What do I do to join you guys? I've bought a new camera, a new computer, new fast hard drives... all to no avail!

Here are the parameters I am currently facing. My old computer isn't fast enough to work with either Cineform or native m2t smoothly. Thus I bought a new computer and transferred the Cineform NeoHDV install. Everything on NeoHDV works as it should except for the tape capture. From reading the Cineform forum on DVInfo.net, it seems like this is a problem related to Vista64 and their software.

On my new Vista64 computer, once the footage is captured everything works perfectly. Capturing HDV footage on that computer however is impossible. HDLink doesn't work. Vegas HDV capture doesn't work. Neither does HDVSplit.

On my old computer, HDLInk doesn't work because my one install is transferred to the new computer. Vegas capture doesn't work because for some reason it divides the capture into a new clip every second or so. HDVSplit works but with drop-outs.

In this particular instance, I lost the transfer from the Z7 compact flash, so I went back to the tape I recorded concurrently. Now I'm back to the old fudging:

What I have to do is capture with HDVSplit with the tab that makes capturing stop on dropped frames unchecked. This will result in a set of crash-prone m2t clips that will only work properly after I move them to my new computer and convert them using HDLink (which works on conversions even if it doesn't work on captures).

It works but it is a royal pain. Yes I know that some of you don't have these problems.

I also know however that there are a lot of people who do, and that those who do have a set of problems with a whole lot in common with my own.

I have read rants on a number of forums where people have just given up in frustration and gone to some other platform. Having just shelled out for a new computer and camera, I really can't afford a new NLE as well, so I will continue to make due.

The other option is just to record to the compact flash card and hope like crazy that I will never need to use the tape backup!
johnmeyer wrote on 4/3/2008, 4:41 PM
My old computer isn't fast enough to work with either Cineform or native m2t smoothly.

Hmmm ... My 5.5 yr-old computer handles native m2t at 29.97 (preview mode) using V7.0d.

Your description of your various problems sounds like some sort of camera driver problem. I can understand Vista choking, because that seems to be what it was designed to do, especially given its limited device support (compared to XP). However, plain XP (i.e., NOT 64-bit) should have no problems.

Also, if you capture with something other than Vegas, I think you are going to have problems. The Vegas capture (if you can get it to work) definitely does something to the HDV file. Bob's idea of error correction being an issue is interesting and, even if it is not correct, is probably getting close to the issue. In particular, all of us have always thought that "capture" was nothing more than a file transfer. Well, I don't think this is true. If you look at the various options in Scenalyzer (yeah, that's DV, and we're talking HDV here, but my point about capture is the same for either), you'll see that it can create a variety of different AVI files depending on what option you select. So, I think that capture applications may in fact "do things" to the data stream before it is written to disk.
Serena wrote on 4/3/2008, 5:19 PM
>>My old computer isn't fast enough ....<<

My old AMD Athlon XP ran both m2t and cineform DIs at 25fps with preview in automode. Slowed with FX applied. In those days I captured from tape with HDLink, not always without having to recapture portions when something went wrong. Later I adopted capturing with Vegas (which was trouble free) and then converting (anyway the machine wasn't fast enough to capture and convert in one operation). Of course that was all with the XP OS and I am still resisting upgrading to Vista.
fldave wrote on 4/3/2008, 6:11 PM
Don't get me wrong, I empathize with you in your troubles, and obviously others with similar issues. Something is causing these problems, and it would be nice to get to the bottom of it. Both to help others having issues, and to keep me from getting them.

I am mainly a "use what Vegas supplies" person, mainly because this is not my main business (yet) thus I think hard before throwing money at other tools. Could I get by with what Vegas/Sony supplied? I found that I could, and didn't buy peripheral software to enhance the core functions/processes.

I can see how your situation is extremely frustrating.
riredale wrote on 4/3/2008, 8:20 PM
Laurence: you tried using a different firewire cable, yes? I had lots of issues with a well-used cable, none with a new cable.
Laurence wrote on 4/4/2008, 8:25 AM
Well I just had some native m2t off tape success!

After weeks of my new Z7 not being recognized by my core2duo laptop, I tried capturing the footage with my older HVR-A1 on the new computer. The HVR-A1 was recognized, but it wouldn't work with the 24p footage. The next time I connected the Z7 to the computer however, it was finally recognized.

I tried the Vegas capture, but just like on my older computer, it would divide the Z7 footage into hundreds of clips, each one less than a second long (even with scene splitting turned off).

HDVSplit now worked though, and the resulting m2t capture was entirely free of formatting errors. The interview was spread over two tapes: one master quality HDV rated, and the other a regular Sony DV tape. Captures from both tapes were perfect.

I am pretty relieved. The raw footage in m2t format is a fraction of the size it would be in Cineform codec avi.
johnmeyer wrote on 4/4/2008, 8:53 AM
I tried the Vegas capture, but just like on my older computer, it would divide the Z7 footage into hundreds of clips, each one less than a second long (even with scene splitting turned off).

I sometimes get these unwanted splits, usually at the beginning or end of the capture. I've never seen as many of them as you have, although I have seen posts from others who have reported getting dozens of these random splits, even with scene detection turned off. It sure sounds like something is interfering with the smooth, continuous flow of data from the camera to the computer. The bad Firewire cable, as already mentioned, is a very real possibility. Also, even though you have had problems on multiple computers, the fact that you have now had some limited success on a different computer calls into question how you have your computers configured. My suspicion always goes to anti-virus software, although there are many, many other programs which install in the background and can suck cycles.

I just did some work for someone yesterday on a Dell laptop. It is the identical model to the one I own, but it was 10x slower than mine. That's not an exaggeration: even opening Explorer took 15-20 seconds whereas it happens instantly on all my computers. I looked at Task Manager and was appalled at all the junk running in the background. Anyway, the point is that you might want to fire up MSCONFIG (just click on Start -> Run and then type MSCONFIG and press Enter). Disable all the startup programs and then try capturing. This is completely and easily reversable, so there is no harm or problem in doing this.

craftech wrote on 4/4/2008, 9:29 AM
Laurence,
How bad is the manual focus on the Z7U that I read about? I read that when on a tripod there is no "push to focus" and that you have to physically move the focus ring forward then back again to do a quick focus when in manual mode. Sounds intolerable on a tripod.

John
cocacolaman wrote on 4/4/2008, 10:02 AM
First, I'd like to publicly thank Laurence for educating me, in a series of emails we have exchanged, about how Vegas handles HD material and for suggesting several workarounds to the 8b crash problem.

I've had the same problem many others have had with 8b crashing. I've been emailing back and forth with Sony support for about a month with no resolution at this point.

I did the following test last night. I have about 4 hours of clips captured from four tapes (from a Canon HV20) with both HDVSplit (these were the clips I originally captured and used in 8b that keep crashing) and the Sony capture utility. Anti-virus software was shut down for all the captures and the same firewire cable was used for all captures.

1. Add the first hour of clips captured in HDVSplit to 8b...no problem
2. Add the second hour of clips captured in HDVSplit to 8b...no problem
3. Add the third hour of clips captured in HDVSplit to 8b...error message, crash
4. Restart 8b, add the same third hour of clips to 8b...no problem, so it doesn't look like it was a bad clip problem
5. Add the second hour of clips captured with HDVSplit to 8b...error message, crash
6. Restart 8b, add the second hour of clips captured with HDVSplit...no problem...again, doesn't appear that a bad clip was the problem (unless 8b has some threshold for handling bad clips)

7. Restart 8b, add all four hours of clips (127 clips) captured with the Sony capture utility...no problem!!!
8. Render about an hour and a half of the timeline to Mpeg2...no problem.

Peter
Laurence wrote on 4/4/2008, 10:10 AM
>How bad is the manual focus on the Z7U that I read about?

It works fine. I just posted on this subject http://www.sonycreativesoftware.com/forums/ShowMessage.asp?MessageID=586824&Replies=0here.[/link]
Laurence wrote on 4/4/2008, 10:15 AM
7. Restart 8b, add all four hours of clips (127 clips) captured with the Sony capture utility...no problem!!!

That makes it really sound like a capture problem rather than a problem with the data on the tape like I originally thought doesn't it.

Glad to see you back in business Peter. :-)
Wolfgang S. wrote on 4/4/2008, 11:48 AM
Laurence,

will not help you - but I did not have any HDV problems with either a 3.2 Ghz P4, but also not with my newer Q6600. I edit native m2t only - and it works fine for me. I work with my old FX1 since the beginning of HDV. I capture only with Vegas 8, but in large pieces - and that works fine for me (I use another tool to generate scene seperation, but that works only for 50i, not for 60i).

Desktop: PC AMD 3960X, 24x3,8 Mhz * RTX 3080 Ti (12 GB)* Blackmagic Extreme 4K 12G * QNAP Max8 10 Gb Lan * Resolve Studio 18 * Edius X* Blackmagic Pocket 6K/6K Pro, EVA1, FS7

Laptop: ProArt Studiobook 16 OLED * internal HDR preview * i9 12900H with i-GPU Iris XE * 32 GB Ram) * Geforce RTX 3070 TI 8GB * internal HDR preview on the laptop monitor * Blackmagic Ultrastudio 4K mini

HDR monitor: ProArt Monitor PA32 UCG-K 1600 nits, Atomos Sumo

Others: Edius NX (Canopus NX)-card in an old XP-System. Edius 4.6 and other systems

John_Cline wrote on 4/4/2008, 12:52 PM
(I just posted the following message in the other curent thread about HDV issues.)

I have had very good luck using a freeware program called "MPEG2REPAIR" to fix problematic .M2T files. It virtually eliminates issues with files that have a tendency to cause Vegas to "hang up" when importing. This includes issues 1,2 & 3 mentioned by Laurence above. MPEG2REPAIR also generates a text file with information about problems it has encountered and fixed in the files. This report text file may provide some clues about why some files work and some don't.

http://www.videohelp.com/tools/MPEG2Repair

VideoReDo also has a transport stream "fix" utility that works well.

John
cocacolaman wrote on 4/4/2008, 2:27 PM
>7. Restart 8b, add all four hours of clips (127 clips) captured with the Sony >capture utility...no problem!!!
>8. Render about an hour and a half of the timeline to Mpeg2...no problem.

>That makes it really sound like a capture problem rather than a problem >with the data on the tape like I originally thought doesn't it.

I left out one piece of information in my earlier post...the clips captured with HVSplit that crash in 8b work with no problem in 7d. So it does not appear to be a capture problem. It appears to be an HDVSplit - 8b - maybe my computer configuration problem.

At their request, I had sent Sony support one of the clips I captured with HDVSplit. I received the following message from them today:

"Thank you for writing back. You shouldn't have any problems with this video once it's captured through Vegas' internal capture utility. We are looking into what possible conflicts may be causing the issue with clips captured in HDVSplit."
John_Cline wrote on 4/4/2008, 3:47 PM
One thing that has always been a curiouslity to me is that HDV clips captured via the Vegas capture utility or HDVsplit or rendered by Vegas are handled by the "m2tsplug.dll" and are identified as "Sony M2TS." HDV files modified with any other software are handled by "mcplug.dll" and are identified as "MainConcept MPEG-2" files. I don't know what this means, but there is something in the header of the file that makes it different somehow.
ushere wrote on 4/4/2008, 8:12 PM
from my first year with v1p and m15 i have to report the only problem i've come across of any significance was print to tape:

i capture with vegas (NO scene split), edit m2t on the timeline, add whatever's required, ie, fx, cc, (all at 8bit) and in general, get fine preview at auto setting (it's obviously NOT best quality full frame, but i know what it's going to look like, so what the hell)....

however, when printing back to tape i (used to) edit with 'no recompress', and found i couldn't print properly, if at all. now, when i'm sure i've finished the project, render to all the various outputs i might need, i render the timeline out back to m2t WITHOUT 'no recompress', and print to tape is fine.

obviously there's some goings on with 'no recompress' that creates a none compliant m2t file that's fine for everything BUT print to tape.

i looked into cineform, intermediates, etc., but in all honest, for the work i do, it's simply overkill. for chromakeying, heavy duty cc, and the like it probably helps, but having seen my end results against similar work done through intermediates, i don't think it's even worth considering.

heck, i was told you can't go more than 3 gens of beta sp with seeing 'major' deterioration. well, component editing, and multi generation editing were everyday occurrences, and yes, maybe after 5 gens things got a bit softer, but then again, the client was happy enough to have multilayered images, and couldn't even see the 'deterioration' i was pointing out half the time....

leslie
NickHope wrote on 4/5/2008, 11:29 AM
>> One thing that has always been a curiouslity to me is that HDV clips captured via the Vegas capture utility or HDVsplit or rendered by Vegas are handled by the "m2tsplug.dll" and are identified as "Sony M2TS." HDV files modified with any other software are handled by "mcplug.dll" and are identified as "MainConcept MPEG-2" files. I don't know what this means, but there is something in the header of the file that makes it different somehow. <<

I was also puzzled by this. Before Vegas could properly smart render HDV I used to smart render using Womble MPEG Video Wizard and the rendered files were identified as "MainConcept MPEG2". Someone in the forum suggested it may be that transport stream files are identified as Sony MT2S and program stream files are identified as "MainConcept MPEG2".
NickHope wrote on 4/7/2008, 10:46 PM
More evidence here that these problems are hardware related.

I've just done a project in native m2t HDV captured with HDVSplit. It's 40 minutes long with 2 video tracks and 3 audio tracks.

My new Dell Precision M6300 (4GB RAM, XP 32-bit, /3G switch) handles the project with no problems. However the very same project on my Q6600 quad core (4GB RAM, XP 64-bit) I'm getting crashing with exception errors, crashing with no error messages, and red thumbnails and previews. It's all exactly the same media and project settings as the project that runs fine on the laptop.

To get the project to DVD on the desktop I had to split it into 2 x 20 minute projects and render 2 x MPEG2 files. Then it behaved fine. Not much point submitting to support as the OS is unsupported.