Z1 cam / P4 3.0 workflow...?

FuTz wrote on 5/25/2007, 6:17 PM
I'll soon be shooting and editing for the first time with the Sony Z1 cam and since I got this already "old" P4 3.0 Pentium computer, what would be your suggestions for a good and efficient workflow, I mean from capture to editing to rendering ?

I did some search using Z1+P4 3.0 workflow but couldn't find something precise about these basic needs (maybe I used wrong keywords).

Thanks in advance for suggestions and feedback.

Comments

PeterWright wrote on 5/25/2007, 7:07 PM
If your PC can't handle m2t files fluently, you have two choices:

Render to "Intermediate" files - there are HDV intermediate templates under avi in Vegas 7, or you can purchase Connect HD from Cineform. The "Intermediate" name is misleading, since these files, which take up about 3 times the space of HDV, can be used directly for final renders.

Render to widescreen DV avi "proxies" . These temporarily take the place of the hdv files on the timeline, and allow fluent editing on slower PCs. When editing is finished, they are replaced by the original HDV files for rendering. By far the easiest way to accomplish this is with Gearshift from VASST - it makes the swapping back & forth a simple click job.

MH_Stevens wrote on 5/26/2007, 6:39 AM
Second what Peter said. I use Cineform, better codex than the one in Vegas. Capture with scene split. For DVD render to MainConceps for DVDA widescreen, or make 760p mpeg2 (.m2t) for computer
johnmeyer wrote on 5/26/2007, 7:24 AM
If you have Vegas 7, you should be able to handle m2t. I have an even older single-thread, single-CPU, single-core 2.8 GHz P4. I edit HDV all the time. I published my workflow 18 months ago here:

My workflow for HDV to SD projects

and received a lot of general HDV workflow answers here:

HDV Questions

While much of this focuses on producing SD from HDV, I still have yet to have a single request for HD delivery on any disc format (even on DVD, which of course, can be done). Thus, the SD component of the workflow is possibly important to you as well.

NickHope wrote on 5/26/2007, 10:53 AM
I can't edit m2t HDV files on P4 3.0C machine in Vegas 7 at all smoothly. It's extremely stuttery, sometimes only playing like 1 frame a second or less and even with preview quality preview. Hence I'm using Gearshift and proxies.

I've done most of the tweaks I should have done, mostly based on the tweaks on the www.videoguys.com. I wonder why my machine can't handle them and John's theoretically slower machine can. Are there any checklists online of things to check? Maybe my PCI FX5200 128Mb graphics card is not good enough?
FuTz wrote on 5/27/2007, 6:12 AM
Thanks everyone.
Seems easy from there, I'm eager to try some tests to find out about my personal issues though ;)
From my understanding now, using a third party app to produce these work (proxy) files will give me better encoded files so I can work with cleaner files ? Gearshift giving me, on top of that, an efficient "switching" capability to go from proxies to m2t files.

Apart from these useful tools, anybody tried from within Vegas itself, like rendering m2t files to AVI (+or- compressed at your taste) ? Were there issues with timecodes matching, etc... when rendering in the end from that "proxy" timeline with m2t files to produce a final HD render?

The fact that Vegas will efficiently use these intermediary files that are 3 times the weight of m2ts is, I supposed, simply a question of what codec is used, which Vegas drives easier than m2ts ?
NickHope wrote on 5/27/2007, 10:52 AM
Your understanding in your first paragraph is correct. You are rendering ultimately from the original quality file, just using the proxies like preview files.

You 2nd paragraph: Gearshift is working fine for me. Rendering a project just now that I edited with the proxies and switched just before render. The clips all matched up as expected. One thing I did mess up was putting my titles in while I had project settings at DV 16:9. They didn't up-res when I changed the project resolution to HDV. Should have put them in when project settings were at HDV to get the full resolution.

3rd paragraph: The Cineform files are bigger because the m2t files have temporal compression. You're only getting a full I frame every 12 frames and then the next frames just describe the differences. That's why they take a lot of processing power to edit smoothly. Whereas the Cineform gives you a full frame every frame.
Laurence wrote on 5/29/2007, 10:16 AM
I was getting a lot of red frames on the timeline and constant crashing with Cineform. On top of that, Cineform performance was nowhere near useable. I tried every method of uninstalling and reinstalling both Vegas, Connect HD and Neo as well as cleaning things up with System Mechanic. Nothing worked.

Anyway, a clean reformat and reinstall of everything from Windows XP to Vegas and Neo changed everything. I can now preview Neo HDV projects with 30 to 40% CPU usage. I have no idea why this helped so much, but it was the only thing that worked.
NickHope wrote on 5/29/2007, 10:41 AM
Right well thanks for that Laurence. It's been ages since I reinstalled for scratch so I'm going to try it and see if it helps.
johnmeyer wrote on 5/29/2007, 11:00 AM
I wonder why my machine can't handle them and John's theoretically slower machine can. Are there any checklists online of things to check?

OK, here's the checklist.

1. You must have Vegas 7. Earlier versions won't edit m2t with any speed at all, no matter what computer you have (but you probably knew that).

2. Here's the secret: Not all m2t files are the same. Not even close. I can't explain why, but sometimes Vegas doesn't capture m2t files correctly. And, if you use a third-party application, like HDV Split, you may get a flavor of m2t that Vegas doesn't like (although I have not yet used HDV Split, and that may not be true).

OK, what does this really mean? Well, to get the performance in Vegas 7, Sony developed some sort of change to their capture application, and the resulting files are somehow "different." I don't know exactly what makes them different, but Vegas sure knows. To tell whether you have the "good" m2t files or the "bad" m2t files, use Vegas explorer (NOT Windows Explorer). Right click on one of your m2t files and select properties. Scroll down to the bottom of the properties listing and at the very bottom, you should see a "Plug-In" section. For "good" m2t files, it should look like this:

Plug-In
Name: m2tsplug.dll
Folder: C:\Program Files\Sony\Vegas 7.0\FileIO Plug-Ins\m2tsplug
Format: Sony M2TS
Version: Version 1.0 (Build 357)
Company: Madison Media Software, Inc.


The key is the name (m2tsplug.dll) and the format (Sony M2TS). The "bad" files will list the "MainConcept" in the Format section and have some other DLL listed. If you see this, then 1-2 fps is all you're going to get.

The other thing is preview resolution. This always makes a difference, but especially with m2t. I just put a "good" m2t on the timeline on my 2.8GHz single CPU/Core/Thread P4. I set the project properties to the HDV 1080-60i preset.

I then shrunk down the preview until the Display setting (shown at the bottom of th preview window) showed 384x216x32.

I played the video (no fX or anything else to slow it down). The fps counter in the lower right of the display showed steady 29.97.

I then repeated this for other preview resolutions and got this table of fps measurements (all at 384x216):

Draft: 29.97
Preview: 29.00
Good: 20.80
Best: 11.8

I then enlarged the preview window to 640x360 and repeated the measurement:

Draft: 27.0
Preview: 25.0
Good: 19.3
Best: 10.5

With your faster computer, you should be able to sustain 29.97 at both draft and preview, at both of the above preview sizes. Of course once you add fX, compositing, 3D, video bus envelopes, speed changes, etc., etc., then all bets are off, just as is the case with SD video.

Hope that helps!


NickHope wrote on 5/29/2007, 12:34 PM
Well thanks for that John. Once again extremely helpful information.

But in my case it's horrific reading and I can't believe I'm reading it now for the first time.

My entire archive of nearly two thousand underwater HDV clips is in the wrong flavour of m2t for Vegas. No wonder my machine seems much slower than other people's.

All my clips have been captured with HDVSplit (mainly for the file-naming) and then trimmed/encoded with Womble MVW (for the smart-rendering).

I wonder if trimming/encoding a Vegas-captured m2t in Womble will preserve the file as a Sony M2TS format file or turn it into a MainConcept MPEG-2. I'll test that tomorrow along with some fps comparisons to yours.

Now I wish I'd taken the hit on storage space and archived in Cineform AVI instead of m2t. But reading other recent threads, that route doesn't exactly sound like a bed of roses either. Anyone else sometimes wish HDV had never been invented?
Laurence wrote on 5/29/2007, 2:36 PM
On the subject of preview resolution remember that scaling is more difficult than showing a big image.

My favorite preview setting is preview at half size with no scaling whatsoever. I turn off the function that scales the video to the window and also the function that scales the video so that it looks like 16:9 rather than 4:3. The image that you see this way is a 16:9 image squished into a 4:3 frame at half it's actual size, but it plays back beautifully with no dropped frames except at the transitions and when I use filters or text overlays. I get smooth playback with a nice big preview window this way on both my studio P4 3.06 and my P4 2.8 laptop.

This brings me to another subject which I've already started several threads about: That is that the "selectively prerender video" function could very easily be enabled on smart-renderable Cineform codec projects, and that if it was, the actual workflow on older generation P4s would be GREATLY improved.

For anyone who doesn't know what this means, here's the quick synopsis: Vegas has a function that was put into place back when PCs could barely handle SD editing. That was that it could go and selectively render temporary proxies of anything that didn't smartrender within a project. That would include all the transitions, titles and filters. If anything changed, the next time you ran the selective prerender, it would render new proxies from the point of the last change. Because these proxies only included the parts that wouldn't smart-render, they were actually generated pretty quickly, and after you ran the selective prerender, you could preview the whole project perfectly smoothly, even on a machine that was only up to handling a single stream of video.

Anyway, this function doesn't work properly in HDV with Cineform, but it easily could. What happens now is that if you run the selective prerender on an HDV project, it renders the whole thing rather than just the parts it needs to. If selective prerender was enabled on HDV projects, it would make working with Cineform on a P4 class machine much easier, even on complex projects.
NickHope wrote on 5/30/2007, 1:05 AM
John did you use (full) or (auto) for your tests? This makes a big difference here, especially on "best" setting.

Will publish results later but initial interesting finding is that for a Sony flavour m2t in a half res window (960 x 540 x 32) I get:

scale video off, best (full) = 25fps preview (I'm shooting 50i)
scale video on, best (half) = 10.3fps preview

In other words it's better in my case to scale video with "scale video to fit preview window" rather than "best (half)" or "best (auto)". Laurence you might want to try that comparison. This might not be true for settings other than "best" though. Note so far I've only tested with "simulate device aspect ratio" on.
Laurence wrote on 5/30/2007, 6:07 AM
Believe me, I've tried all the different preview resolution configurations!

What works best for me is preview(half) with scale to window and simulate aspect ratio both turned off.
NickHope wrote on 5/30/2007, 9:19 AM
Well, optimum video preview settings appear to depend on whether you have Sony or Main Concept flavour m2t files on your timeline.

OK, going on from John's tests I've been testing all day and here is what I've found:

1. HDVSplit is generally capturing "Sony" flavour m2t files (i.e. fast ones). However in my case it occasionally captures a "Main Concept" flavour m2t file (slow one) but I think this is related to lost packets which are caused by breaks in timecode caused by running my Z1 in "Quick Rec" mode. Also the first file captured off tape by HDVSplit is a "Main Concept" flavour one.

2. Trimming a Sony flavour m2t file in Womble MPEG Video Wizard will turn it into a "Main Concept" flavour m2t. So although Womble smart renders the file and preserves the full quality of the main body of the file, it costs in terms of performance in Vegas 7.

3. Here's a table of frames-per-second video preview results for various settings I've tried. I'm assuming that John had (full) set for his tests. Also note that I'm working with 50i material, not 60i.


384 x 216 x 32
Meyer * Hope Hope Hope
P4 2.8 Desktop Desktop Laptop
60i 50i 50i 50i
Sony m2t Sony m2t M/C m2t Sony m2t

draft (full) 29.97 25 24-25 25

preview (full) 29 25 19 25

preview (auto) - 25 6-25 25

good (full) 20.8 25 19 25

best (full) 11.8 25 19 25

best (auto) - 19 8 19
(sim AR on)
best (auto) - 19 8 19
(sim AR off)

640 x 360 x 32
Meyer * Hope Hope Hope
P4 2.8 Desktop Desktop Laptop
60i 50i 50i 50i
Sony m2t Sony m2t M/C m2t Sony m2t

draft (full) 27 25 24-25 25

preview (full) 25 25 17 25

preview (auto) - 25 8-21 25

good (full) 19.3 25 14 25

best (full) 10.5 25 17 25

best (auto) - 17 8 16-17
(sim AR on)
best (auto) - 24 9 16-17
(sim AR off)

960 x 540 x 32
Meyer * Hope Hope Hope
P4 2.8 Desktop Desktop Laptop
60i 50i 50i 50i
Sony m2t Sony m2t M/C m2t Sony m2t

draft (full) - 25 6-18 -

preview (full) - 25 8-12 -

preview (auto) - 25 9-21 -

good (full) - 25 10-11 -

best (full) - 25 12-13 -

best (auto) - 10-11 3-4 -
(sim AR on)
best (auto) - 11-12 4-6 -
(sim AR off)


My desktop is P4 3.0C, 2Gb RAM
My laptop is Pentium M 1.86, 502Mb RAM


4. Ripple editing on the timeline was much "clunkier" at all settings with the Main Concept m2t files than with the Sony m2t files. By which I mean more of a delay until the video preview showed the correct frame that was being dragged to.

5. Turning "scale video to fit preview window" off did not improve playback smoothness in any of the tests.

Conclusions (for my case):

1. Turning "simulate device aspect ratio" off doesn't make enough difference to outweigh the squashed preview.

2. On a timeline of Sony flavour m2t files it's best to run with "best (full)" preview and "scale video to fit preview window" turned on. Can still hold 25fps preview even at half HDV resolution.

3. On a timeline of Main Concept flavour m2t files it's best to run with "preview (auto)". Smaller preview resolutions play smoother but in general performance is just too clunky and inconsistent to edit with.

John, it's surprising that your fps at "best" dropped so sharply while mine stayed at 25fps. Something else at play perhaps?
FuTz wrote on 5/31/2007, 5:41 AM
Boy oh boy, this SURE is some information I'm gonna keep for future reference. Wow, thanks guys !
NickHope wrote on 6/1/2007, 1:20 AM
For reference, here's a previous thread on the issue of Sony vs Main Concept m2t files:

http://www.sonycreativesoftware.com/forums/ShowMessage.asp?ForumID=4&MessageID=526083
FuTz wrote on 6/3/2007, 4:17 AM
Bottom line: I'm happy I upgraded from V5 to V7..! :)
Thanks for this additionnal link Nick, I just bookmarked it with "m2t files handling" title ;)
FuTz wrote on 6/10/2007, 4:22 PM
Another question: I've seen that HD files have the same weight in terms of GB that DV files have , ie around 13GB/hour.
How come it's so light? An editor I know (FCP user) goes up to 60GB/hour with HD files vs. 17GB for DV. Is there an explanation for this? Codecs, compression etc..? (I know FCP uses only Quicktime files and that they're heavy...)
NickHope wrote on 6/16/2007, 6:35 AM
With DV you're getting a full frame of video every frame. Each frame has spatial compression to save size. The HDV files that are a similar size (actually mine are smaller than DV) are.m2t files and have temporal (time) compression as well as spatial compression. You're only getting a full frame ("I frame") of information every 15 frames (for 60i) or every 12 frames (for 50i). The other frames just describe the differences from that I frame. Your FCP editor friend will be getting 60GB/hour after uncompressing to an intermediate codec that gets rid of the temporal compression and gives him a full frame every frame. In Vegas such a codec might be the Cineform AVI codec.

I'm no expert but that's my layman's take on it anyway. There's more to it such as B frames and P frames. Try a Google on "temporal compression".