HDV, Firewire, RAID and Ultimatte

vicmilt wrote on 9/24/2006, 4:51 AM
For years now I've been yappin', "Don't get HDV until you can deliver it, or have a client who will pay for it". Of course, I've drooled over those HD images from afar, but held off nevertheless.

Now (at last) I have a need to shoot and deliver HD footage and so I'll be looking for the experienced help on this site. Experience beats theory every time.

So - to my first question -
my normal work procedure in SD is to capture and edit on outboard firewire HD's. That way I can use one drive for each client or job. Done it this way for years w/o problems.

As I gear up for this HD project:
Will the Firewire HD's continue to work correctly?
Do I need a RAID array to do "all the usual stuff" - color correction, slo mo, 4 to 6 tracks of video.
And since I am expecting to source about 80 hours of video at a minimum , to deliver a 75 minute end product - what is the best workflow procedure?

Thanks in advance,
v
ps - anything else I should be aware of in this monumental shift?

Comments

farss wrote on 9/24/2006, 6:25 AM
The firewire drives will continue to work as always. They're just devices for storing files. That those files contain video instead of spreadsheets matters nothing at a basic level, the drives and the OS are none the wiser.

HDV is the same datarate as DV25, so pretty much what works for SD will work for HDV, just finished capturing HDV down a firewire connection that also had a firewire drive on it that I was captruing to, no problems.

Now comes the bad news.

HDV is way more CPU intensive to playback. V7 seems to have improved matters but I don't think things will be as fluid as DV, not with several tracks.

As you're probably aware the best approach is to use the CF DIs. Now they're bigger files but use less CPU overhead. Bigger files mean higher data rates and as you add more tracks that data rate goes up. Playing back 5 tracks of composited video does mean 5 times more bandwidth needed from the drives.

So probably a faster bus than firewire would help, eSATA is the way of the future, it's much faster and cheaper than firewire.

Also 80 hours of source converted to CF DIs is a fair slab of data!

None of this is outrageously expensive, it's still inside the scope of off the shelf bits and pieces.

However if you want a full screen preview for serious image tweaking for say CC you could start to blow serious dollars on monitors. Or you could farm that part of it out.

What is the deliverable format. HDV or HDCAM?

Of course there's many issues to do with shooting HDV but as you're more than used to shooting 35mm you've got the skills to handle HDV. That puts you in good stead for making the transition, probably more of a step back into familiar territory actually.

Bob.

Bob.
FuTz wrote on 9/24/2006, 7:07 AM
I'm still not into HD at all but would 10K rpm drives improve something over 7.2K ones ?
JohnnyRoy wrote on 9/24/2006, 7:45 AM
> what is the best workflow procedure?

As farss said, same as DV now that Vegas 7 has considerably improved raw HDV performance on the timeline (it’s better than CineForm files). If you have a lot of tracks and you can’t get good playback performance, you still might want to use something like GearShift and work with proxies.

> I'm still not into HD at all but would 10K rpm drives improve something over 7.2K ones ?

Nope. I posted my findings at the bottom of the HDV page on my web site. The 10K drives cost 5.4x more and only give a 12% boost in read speed. My 500GB RAID of 7.2K drives costs me $200. A $500GB RAID of 10K drives would cost $1,097!!! Hardly worth it. I doubt you would feel a 12% difference.

~jr
vicmilt wrote on 9/24/2006, 9:11 AM
So JohnnyRoy...
Would you say that GearShift makes the most sense for me, taking into consideration that I'm gonna have a LOT of original footage to go through?
What is the "transmogrication" rate of lets say an hour of original to the "correct - by your def" GearShifted proxie. That is how long should I allocate for "shifting"?
Storage space?
And are there any glitches... that is, things you should NOT do with the shifted proxies, because they won't rebuild correctly?
And what do you think would be the "dream" editing setup for a large gig like this?
If all this has been gone through already, please point. My searching came to no avail. And with the new V7 - how have things changed?
v
fldave wrote on 9/24/2006, 9:31 AM
Vic,

If I were to build a new PC today, I would base it around the Core 2 Duo, specifically the E6600. Not the top of the line chip, but a screaming combo of price/performance, leaving room to upgrade later.

http://reviews.cnet.com/Intel_Core_2_Duo_E6600/4505-3086_7-31973857.html?tag=pdtl-list

7200 rpm drives are fine, the bottleneck with HDV is the cpu decoding, which Sony has greatly improved lately in V7. With 80 hours of footage, you'll need a lot of HD space with Cineform intermediates.
DataMeister wrote on 9/24/2006, 12:41 PM
Does anyone know, or can they theorize, on if the HDV decoding is in separate thread for multiple composited tracks. In other words, would a quad core Xeon setup like the Mac Pro handle more simultaneous HDV tracks than a standard Core 2 Duo system. Or does the decoding playback sit in a single thread?

And to take it a step futher, would dual core Xeon chips on a 4 socket motherboard (eight cores) work even better with multiple tracks?
farss wrote on 9/24/2006, 1:15 PM
Victor,
I think perhaps one of the most important question is:
"what do YOU need to creatively edit".
Do you need full frame, full res playback in real time. Can you make the right creative decisions from a small, low res preview or do you need to see the full image. In other words, how much of the "feel" of the shot do you need. From my limited dabbling in the creative side my feeling is that there's a difference between editing SD and HD. If you're simply editing an event in HD where pretty much what you've got is all you've got then it doesn't matter much, you're just going to have to go with how the shot was.
However if you've got lots of footage and you're making decisions based on the look of the shot then you might have very different needs.
Or your workflow might encompass this as well. Maybe you want to playback you tapes from a deck to a full res monitor and log your shots. Again there can be some traps with this workflow and HDV.

Bob.
fldave wrote on 9/24/2006, 1:45 PM
Vic, a few storage stats, from a 10 sec sample. Conversion times based on P4, 3.2 HT machine.

m2t: 33MB; 10 Sec clip

Cineform Intermediate (1080-60i) - 234MB; took 28 seconds to render
SD NTSC Widescreen, Best: 37MB; 42 sec to render
SD NTSC Widescreen, Good: 37MB; 24 sec to render

I could see some difference between SD Good and Best, but not very much zoomed in.

Multiply the above by 28,800 for 80 hours HDV video.
Jay-Hancock wrote on 9/24/2006, 1:49 PM
fldave - I assume you are rendering all of these from a Vegas timeline? Do you have HD Connect? It can render a .m2t to all those formats you mentioned. I have no idea what speed it handles these conversions with.
fldave wrote on 9/24/2006, 1:59 PM
Don't have Connect HD, figure it would be a must with that much video. Those are render times from Vegas timeline.
Jay-Hancock wrote on 9/24/2006, 2:14 PM
Victor:
Maybe a little unrelated, but since you are looking into HD you are undoubtedly looking at ways to playback your HD productions on a TV.

If you don't have a deck or HD/SDI or you wish to playback on a TV without using the camera, there is a $249 player on the market that will play HDV .m2ts. It's called the LinkPlayer2, by a company called IO Data.

It's essentially a DVD player. You can hook this thing up to an HDTV with component video and playback your high-def files in .m2t format or WMV-HD or DIVX-HD. For .m2t the max bitrate of 19Mbps and audio at 192kbps, somewhat less than what comes out of a Z1 camera thus a re-encode is necessary. But the picture looks good and for $249 it's a quick solution. The box has an ethernet port which you should connect to the Internet and download firmware updates. You can also use it to stream the video files off your PC. Or you can simply burn an .m2t file to a DVD and put the disc into the player. I stream it from my PC via ethernet. To me the big advantage is you can see what your end result will look like on a television without having to print to tape (and I don't have a deck).

They also have a more expensive version ($399) which has an HDMI output and has upconversion for SD DVDs. And I know there are a few competitor products out there that do essentially the same thing.

You still don't get DVD-type authoring with this setup (I don't know that DIVX allows this), but neither do Blu-Ray & HD-DVD, and this thing doesn't cost you $1k+.
vicmilt wrote on 9/24/2006, 4:14 PM
Guys -
this is great - I knew I'd get the right feedback...

so what is Connect HD? Is it software or hardware? and where does it come from. I'm thinking that a lot of folks are going to read this thread eventually and I'll play the "newbie" because when it comes to HDV - I am - hey, we ALL are.

Farss, I'm thinking that since I will be redigitizing the footage in the "end game", a quick transferring, low overhead medium resolution proxy is probably the way to go. Then when I ease into the last generations of editing, I'll uprez and work out the additional equipment necessary. Whether a purchase, oursource, edit shop or whatever will come in the proper time - so what do you see as the proper equipment for me for the present? I'm looking into weibetech.com SATA docks as an upgrade for the outboard Firewire docks I've been using.

Jayster - I love the idea of an outboard player for the M2t files. I had no idea one even existed - thanks. (yikes - now I've got to get serious about an HD screening solution... but not this week.

I'll keep you in touch with my progress.

v
Jay-Hancock wrote on 9/24/2006, 4:25 PM
Vic:
Connect HD (got it reversed in my earlier post) is Cineform's software solution that converts HDV from MPG2 into a "virtually lossless" AVI format. You know well in SD terms the difference between editing with MPG2 vs. DV. Same thing applies here. The Cineform codec doesn't suffer from generation loss as does HDV. . In a lot of ways it's better. It uses a 4:2:2 color space, for one thing.

The Connect HD also does (pretty darned quickly) some great things, like resizing video (SD to HD and vice versa), deinterlacing, and pulling out telecine / pull-down. It also allows the captures from a variety of cameras that could never have worked in Vegas to suddenly be compatible.

If you have any more questions now or later about the LinkPlayer, feel free to email me and I'll answer whatever I know.
bruceo wrote on 9/25/2006, 11:27 AM
If you have 80 hours of footage do NOT use Cineform. V7 actually handles HDV MUCH better than Cineform intermediate. Also your drivespace requirements are 11GB/hr vs 40+ with cineform if you want to use smartrender.

I would only use cineform if I was in a multiple generation rendering workflow.

Currently I am using Cineform's capture utility for capturing the m2t stream because Vegas' is subpar and riddled with some kind of problems.

If you need any specific feedback I can give you plenty of answers because I have worked on proxy, gearshift, cineform, & native HDV workflows on several computers over the last 18 months on over 350-400 hours of HDV footage.
Steve Mann wrote on 9/25/2006, 12:44 PM
Jayster:

From what I've seen of the LinkPlayer2, I am impressed. Our local Fry's has them on their demo HD Displays (not for sale, they are on loan from the HDTV distributor's) and the images are awesome. I don't know if they are Divx-hd or wmv-hd (and asking a technical question at Fry's is like asking your plumber how to do brain surgery), but it does provide us a way to deliver HD.

I don't see in the specs where it can play an M2T file. Have you tried it?

Steve M.
farss wrote on 9/25/2006, 2:56 PM
Don't know a bout the LinkPlayer2 as IO Data still will not ship them outside the USA, you'd think some enterprising soul would be flogging them off via eBay (hint, hint).

However somewhat to my surprise I found last night that Nero's Showtime plays my mt2 files perfectly on my far from zippy Shuttle PC.

Victor,
I'm concerned about your proposed workflow. The well discussed issues of trying to do an online / offline with Vegas is one thing. Doing it with HDV does seem to be a whole new world of pain on top of that. HDV and timecode just don't seem as solid as they are with DV. I haven't really invested the necessary time to really have all the answers / know all the pitfalls. Newer equipment does seem to handle this better, I've recently found I can write real T/C to HDV tape and read it back and get Vegas to see it. Just how reliable what I did to make that happen is and how reliant it is on the specific kit and software used to do it I don't know.

Unless someone else can chime in with a 100% certified workflow my best advice is test, test and test again. Right from the camera through to the system that'll do the online. The only 100% certified workflow that I know of is rather expensive. The client is using a HD Connect LE to dub from HDV to HDCAM SR and doing offline / online from that. If you're interested I can get you the specific details, their final output does look pretty impressive but they've got pretty deep pockets. No matter what though testing is vital. Having to conform 80 hours of HDV by hand would be both tedious and very expensive.

Bob.
vicmilt wrote on 9/25/2006, 6:14 PM
Deep pockets - no - not on this particular gig.

But I do want all the quality you can wring out of today's HDV climate.

I'm sort of thinking that I'll do preliminary "Selects" editing using the Vegas 7 direct w/o any proxies or intermediaries. Once I've cut the crap out... well it looks like I'll have to decide at that point.

The truth is that 80 hours is probably a conservative number. If anything it'll be more. I can't believe I've got to start ALL OVER AGAIN.

In a way, it's exciting, all this change. On the other hand, it's wearisome having to learn a whole new language and skill set - just when I was getting "good" again. Ya know, film was a pretty solid media platform for over 60 years. Sure they went to accessible color, but you could still cut today on a moviola you bought in 1945.

Well, that was even before my time. But I was pretty confindent in the '70s that I knew pretty much EVERYTHING... and I pretty much did. I wasn't alone. You could learn everything about film. There were three emulsions, ten cameras and two editing systems.

Well, by next month - with this group, the internet and Google - I will once again.

Thanks for all your patience and support. We are all in this together.

v
Serena wrote on 9/25/2006, 7:13 PM
Vic, I went from film to HDV (with one DV project between) and I think you already know 99% of what's important (you already know video and vegas). There is the business of getting the tools right for HDV editing and I'm sure you're going to do a short test project before starting the big one. Be nice to get the tool-learning out of the way before getting submerged.

EDIT: given your depth of knowledge, make that 99.7%. The only change you're only looking at is computer (power & storage&tools) and workflow compatible with that and 80+ hours of takes.
vicmilt wrote on 9/25/2006, 11:27 PM
Serena -
Couple 'o years back, wrote a book titled, "Quit Whining" - www.QuitWhining.net
Now you know why :>))

v
Serena wrote on 9/25/2006, 11:48 PM
Vic, I remember seeing the T-shirts!

Serena
Jay-Hancock wrote on 9/26/2006, 12:17 PM
n19093 - sorry I couldn't get back to you earlier.

Yes, I do use the LinkPlayer2 for playing .m2t files. In fact, this format seems to give the best quality of all the HD formats. Tradeoff compared to the other HD formats (WMV-HD and DIVX-HD) is that they make smaller files. I play .m2ts from a PC via ethernet, but IO Data talks about how to configure the DVD drive to spin faster for those who prefer to play .m2t filed from a burned DVD. This is explained a bit at the bottom of this page about setup.

This player uses a chip from Sigma designs that decodes quite a variety of video formats. When I first bought this device in early 2005 it was an "early adopter" product with bugs and issues. But now they've updated the firmware multiple times and I find it works well for my needs. The DVD drive is noisy but it doesn't bother me. Forum users complain a lot because the player used to upconvert SD on the analog component output and now they removed that feature in firmware, but that had to be shutdown because of legal issues. Not a problem for me, since any HDTV by definition upconverts SD signals so that they can fill the whole screen:-)

.m2ts must be encoded at bitrates within the device's limits. This limit for MPG2 is 19Mbps for video and 192 kbps for audio. Using .m2ts from a 720p camera that would work out of the box, but as you know 1080i HDV 1080i comes in at 25 Mbps / 384 kbps respectively. No problem, I just do a re-encode from Vegas with bitrates that work. I use VBR encoding with 15 Mbps target and 19 Mbps peak and it works great, but maybe CBR at 19Mbps would be ok. I use 160 kbps for audio, but the manual says I can go up to 192 kbps (haven't tried that).

You could just burn discs and use the DVD drive. But personally I prefer to use the ethernet capabililty and stream media to if from my PC on my LAN. For ethernet streaming you should install a (free) 3rd party media server called WIZD which has download links on their forum. If you don't bother it's not a big deal, but I find it works better than IO Data's media server software.

Bottom line, now that they've stabilized the firmware this thing works great as a player for .m2ts (at the bitrates shown in the manual). The picture is awesome, and I get to enjoy my HD productions on a standard HDTV (no decks, no camcorder attached, etc.)

Farss -- the LinkPlayer (thanks to firmware updates) does play PAL for SD output. However, the HD via component video is all at 60i. Aren't your HDTVs all 50i?