Findings with "serious" use of Network Rendering

Liam_Vegas wrote on 3/20/2005, 12:22 AM
Project:
Produce a 10 minute video loop for a trade-show - for display on a couple of 42inch wide-screen plasma displays.

Insane schedule. Changes at the last "hour" (almost). In fact changes required to the loop from one-day to the next of the actual trade show (the show is on-going right now Friday-Sunday).

Problem: Render time for this 10 minute project is of the order of 8 hours. That is on a 3.0Ghz HT box.

The video loop uses masses of tracks (60) / track motion / keyframes / high-res photos (mainly PSD's) with lots of movement all the time.

Solution: Network Rendering to the rescue

I have four machines that I have setup Net Rendering on.
a) my main 3.0Ghz HT machine
b) cheapo AMD 3000+ HP desktop
c) DELL Inspiron 5150 laptop (3.0Ghz)
d) Sony Vaio K37 Laptop 3.2Ghz

Using Network Rendering (Template DV AVI Widescreen Sony YUV codec) takes approx 2hrs 30 minutes. Almost four times faster than with the one machine.

I did encounter many problems along the way... but I just had to find work-arounds (or the solution of course).

Here are some of the challenges/findings.

FONTS

So... my project used an "unusual" font called Copperplate. It was not on my main machine... so I downloaded it and installed it. When I did my initial network rendering I found the text using this font would change (jump) from one font to another. You have probably guessed my mistake. I did not install the font on the other Pc's. Strange this is the font was actually on ALL the other machines... must have come with an upgrade to windows XP or something. However the fonts were different versions. The solution was to re-install the font so they were all the identical version.

BMP FILES

Some of my assets from the client were BMP files. My initial network renderings of the project were generating errors. on one machine. I could not get it working at all. The basic error message displayed in the status window merely said it could not open the Veg file. I found the problem by trying to open the Veg file from the DELL. I immediately received an error about Quicktime Components not being installed. This error was not reported by Net Rendering. I traced the fault to a particular asset that was a BMP file.
I think BMP may need the QT components to be installed. This machine did not have them installed. In the end I just re-saved the graphic as a PSD file and all was well.

DREADED BLACK FRAMES


The BIGGEST problem I had with net rendering was that at a consistent point in the rendering the resulting final stitched video file would have the final 2 minutes or so completely black. This issue remains... my work around was to re-do the network rendering for that final portion... and so adding a further 38 minutes to the total rendering time.

I believe this is a bug... and I will report more detail on this to Sony. This was driving me crazy. I decided to tell network rendering to not delete the intermediate video files that are rendered by each PC . This allowed me to see what was happening more clearly. As the black frames occur across a full two minute segment - these were being generated by ALL the network rendering PC's. It was not a problem with just one or two of them. In monitoring the intermediate video segments as they were being completed by each rendering PC I could see that every single video segment appeared to be OK. I was monitoring them by simply viewing the thumbnails for the videos using Windows Explorer. The window would remain open while rendering was occurring and when each segment was completed windows would dutifully display a thumbnail. My thought was that if Windows could display a thumbnail then why does Vegas not see it. I thought perhaps it was a problem that was occurring during the stitching process. It was not. The segments that made up the final two minutes, when loaded into Vegas by themselves only the audio would be "displayed". The Video was unavailable.

Summary

Network rendering worked (after a fashion) for me on this project. Although I had problems... with the workaround to the balck frame... this undoubtedly enabled me to do this project at all. Without network rendering I would have been unable to meet the deadlines.

Hope that was useful.

Comments

busterkeaton wrote on 3/20/2005, 3:56 AM
Liam,

Thanks for the data, that is very useful.
jlafferty wrote on 3/20/2005, 8:13 AM
Thanks for the reportage -- keep it up!

- jim
BrianStanding wrote on 3/20/2005, 9:35 AM
Thanks, Liam for the report and pointing out some of the "gotchas."

Do I understand correctly, that the black frames appear during the stitching process?

It seems to me the stitching process is the weakest link in Sony's network rendering scheme. Hopefully, this will be improved in Vegas 6. I' ve now taken to splitting my network rendered projects up into arbitrary segments, and rendering with "distribute render among computers" turned OFF.

Then I manually put the two segments together in a new Vegas project. I probably lose some efficiency if one segment has more complicated renders, but I figure I gain it again by avoiding the stitching process. I haven't timed this yet, but it seems to be about twice as fast (two computers) as one computer solo.
Liam_Vegas wrote on 3/20/2005, 5:43 PM
Do I understand correctly, that the black frames appear during the stitching process?

That's exactly what i THOUGHT when I first started investigating this problem. That's because I could in fact see a thumbnail for every completed clip using Windows Explorer as I was monitoring the progress of the rendering. When I eventually got the final stiched AVI file I would see the end 2 minutes would be completely black. Of course my initial reaction was that it was happening during stiching. However... when I told Net Rendering not to delete the temporary files after stitching I could see the problem more fully.

In this case... although Windows Explorer could produce a thumbnail for all the clips that were black in the final AVI file... if you load those clips directly into a Veg file that is when you see that Vegas cannot in fact <SEE> the video.

I have saved these clips so I can do some more investigation on them... and possibly get them sent to Sony. It may also be related to the destination format of Sony YUV widescreen. It seems that even though I render to widescreen format the file always comes into Vegas as being the wrong pixel ratio (.909 instead of 1.2121).... so perhaps there are other issues contributing to this problem.

However I am running 5.0b... and I was loathe to upgrade to 5.0d while this current project was on-going. Perhaps 5.0d will fix this... who knows?

I am always concerned that adding a new version into the mix at the last minute can sometimes be a disaster.

SAVE TIME - DON't STITCH.

The stitchcing process is a bit of a bind... and in the past what I have done is to not bother with the stitch in any case (I just canel out of it). As long as you set "do not delete temporary files" then you can load all the individual clips into a Veg and render out to MPEG directly. This saves all that stitching time. This process is particularly appropriate if your end result will be MPEG... or even if you are simply going to do a PTT.
vicmilt wrote on 3/20/2005, 7:13 PM
Liam - I realize that your project is now probably just a memory but...

First of all - I'm a brute force gorilla - I just do whatever works to get the job done. Since "work always takes the time budgeted to it", we are always a little short-changed at the end of any job with a deadline - that's the way it is.

I had tried network rendering, and found it a little too buggy to totally rely on as a "full force - totally invisible" render farm. When it was good, it was great, when it wasn't, well it really wasn't. I would never commit to an eight hour render - too scary - too fraught with the potential for a totally unrelated machine crash.

Now I TOTALLY understand last minute changes - that's my life. But in a ten minute film there are bound to be vast areas that are "fine" and not subject to revision, and that's where I think you are losing control of your project.

I always break up any edit inot logical sections - whether 10 seconds or 2 minutes is totally up to the amount of work involved in that particular section.
I edit these various segments and render them out, individually.

They all get assembled into a basiclly simple edit which I call "AssembleALL". Because everything in "AssembleALL" is pre-rendered, it's final render time is relatively short. As things get revised, I go back to the original edits (of that segment) make the changes and re-render THAT SECTION ONLY.

I then once again plug the new re-render into AssembleALL. So I'm never rendering huge sections of the video at once. Even a total computer crash doesn't won't hurt me too much. The concept of assembling 60 tracks of video into a 10 minute project is theorectically GREAT, but asking a limited computer setup like the one in Vegas to accomplish this feat is a bit beyond the design specs - for that kind of efficiency, I'd want a quarter of a million doallar AVID Symphony, or somehting similar. The end result won't be one iota Better - but those bucks give you computer POWER for real time or at least very fast rendering.

Vegas (which I use ALL the time) is better suited to littler chunks. This is not a design flaw, it's reality. You can't expect a sub-$1,000 running on a sub $1,000 computer to compete with multi-gazillion dollar set-ups. The nice thing is that the END PRODUCT CAN BE EVERY BIT AS GOOD. You've just got to adjust your work flow to the machinery at hand.

I hope I can convince you to try this way of working. It's very smooth and super easy for those last minute revisions.

best,
v
Spot|DSE wrote on 3/20/2005, 7:31 PM
Having been working with one of Vic's projects recently, I can tell you that his methodology is a little unconventional, but it's organized, and it works.

Having said that, I only use network for longer projects. It's quite handy when you've got a 5 hour timeline. The information you present Liam, is very useful, IMO. Thanks for taking out the time to create it.
johnmeyer wrote on 3/20/2005, 8:15 PM
Liam,

Thanks for taking the time to do this. Make sure you send it to Sony, either as a bug report or in their suggestion box. They seem to be reading the messages in this forum more closely these days, but they might miss this one.

Network rendering is tantalizingly close to being really useful, especially for a project like yours. In fact, your project is the perfect candidate: Short timeline (ten minutes), but horrendous rendering burden. The short timeline minimizes the network traffic.

The suggestion to eliminate the stitching entirely was made to Sony in June last year. Given the VEG structure, there is absolutely no reason that Vegas should need to stitch a project together: just point to the rendered components and then start the encode (to MPEG) or print to tape.
Liam_Vegas wrote on 3/20/2005, 9:38 PM
"v"

Thanks for your suggestions. You are a little in the dark as to the total nature of the project and unfortunately your assumptions about "vast areas that are fine" does not reflect the reality of this project. However what you recommend is indeed a very appropriate workflow and I heartily support the methods you suggest.

I did in fact employ much of the workflow you describe (I always do)... however there was in fact a continuos looping background (Editors Toolkit) and the changes requested by the client required adjusting timings of many sections throughout the video and therefore pretty much the entire video had to be re-done several times - especially on the night prior to the trade-show.

In fact I also made changes over the first night of the trade-show - but as these changes were to a couple of specific sections with no impact on other areas I did render just those parts (two 30 second segments) independantly and re-combine them back into the project in a much more streamlined fashion.

I hope that clears things up for you.
Liam_Vegas wrote on 3/20/2005, 9:44 PM
DSE,

I also would normally use network rendering on longer projects (although 5 hours beats anything I have done o far)... but longer doesn't always mean more appropriate for network rendering. In the case of this 10 minute piece it <really> benefited in view of the processor intensive FX's and other things I included in the project. In the past when I have use network rendering - including in stitching time I ended up getting about 50% improvement. In this case the nearly 4 to 1 improvement was very welcome indeed.

In any case. I appreciate your comments very much.
Liam_Vegas wrote on 3/20/2005, 9:48 PM
John

I also submitted the same suggestion to them last year... probably as a result of our earlier posts about this back then. Hopefully something will come along soon.

This approach for a net render final VEG file would be a perfect use for the accurate rumor of nested timelines within the accurate rumour of the release of V6 perhaps?.
ezway wrote on 3/20/2005, 11:07 PM
A Resource manager on the first box would help in a situ like this.
I would grab all the resourcees in a project then check which boxes came up with all the requested resoures. If it comes back with 8 out of 20
units can run the job without error, you get a report on what unit 14 is missing, and such. Also the Resource manager could time and buid reports for track records, history and exceptions.
Useful Data for everyone.
Marty
JJKizak wrote on 3/21/2005, 6:58 AM
Do all the computers have the same OS? All the same updates? The same version of Vegas in each computer? And is all of this necessary? Just wondering out loud as I have four computers each with XP and Win2k.

JJK
Liam_Vegas wrote on 3/21/2005, 8:30 AM
As far as version of Vegas... yes... you should have the identical version of Vegas installed on each machine. Doing otherwise could be quite problematic (I know for a fact that 5.0a works very differently to 5.0b as far as network rendering).

As for O/S - in my case I have oine PC with XP pro, two XP Home, amd one (would you believe it) Windows 2003 server. The general rule is that every machine does not need to be absolutely identical. However, each machine must have access to the same resources that are called for in the veg file presented for rendering.

This means not just the actual assets in the Veg (Video / stills etc) but extends to fonts, and other software that may not be imediately obvious (such as Quicktime components, fonts, additional plug-ins (NR2, PIxelan etc etc).

The ideal arrangement for Network Rendering is to use identically configured machines (all software, plug-ins, fonts, O/S). In practice that just does not happen... with the result you need to be extra careful about what things are different between them and to correct that before something bad happens.

[EDIT] - added a few additional comments and did some spell checking
rdolishny wrote on 3/21/2005, 2:37 PM
Very useful document I've saved it.

With Lightwave when we encounter black frames it's usually a memory issue. LW network render renders a black frame and reports no error. Subsequent render managers look out for this but the stock 'sreamernet' is very primitive.

For Vegas I'm very happy with "MultiRender" from peachrock.com. I made my investment back on the first day of a 19 day job. It does a lot, but for me I used the ability to render sections of the timeline, even timelines in other instances of Vegas. I would que up sections and it would politely sit and wait for me to press "render" or render on the fly. Whatever I wanted. It was very intelligent and the only crash was soved with a quick email and .a upgrade.

I consider it a very stable "one computer" version of network rendering if that makes sense.

- R
Liam_Vegas wrote on 3/21/2005, 2:44 PM
I agree. Randall at Peachrock really knows how to produce VERY nice code that works extreemly smoothly.

I'm just about to post on the black-frame issue. It is far stranger than I first thought.
vicmilt wrote on 3/22/2005, 2:45 AM
Embarassed to admit that I too have used (forgot to mention) and do here heartily endorss Multi-render.
Works great for multi MPEG files as well as various pre-rendered AVI's as I mentioned above.
With it, I can do the edits, then leave the room for hours, as the computer (finally) does the work unassisted.
Thanks Randall...
v
vicmilt wrote on 3/22/2005, 2:59 AM
Liam -
thanks again for your very detailed and insightful reporting of the network render process - I will give it another try at the first opportunity - and yes, I do understand the nature of "soft sand editing" via ever changing client requests/demands - my heart goes out to you (see below).

those of you who are considering doing this kind of work for a living... please take note of Liam's report - not only of the network situation (which will get better with more software/hardware development) but - more to the point - the client/workflow scenario.

The stress involved in working for pay against deadlines cannot be imagined - it has to be experienced - and it sucks.

I am firmly of the opinion that computers are far more intelligent than we give them credit for, and that they lie in wait for "deadline and rush" situations to crash, burn and otherwise torment their owners.

ALWAYS ALLOW AN EXTRA DAY FOR YOUR COMPUTER TO SCREW YOU UP (you ain't gonna get it - but if you THINK this way - you may not have to spend as much time in the bathroom throwing up) - again - if you KNOW you can have the "job" done on Wednesday - tell the client Friday. If all goes well, and you deliver Thursday AM - you will be hailed as a Genius and "the fastest editor I know".

If you are late - you are an (expletive deleted).
Last time: Never tell the truth about a deadline delivery - always add at least a day for the computer...

For both professionals and wannabe professionals, this information is worth hundreds of thousands of dollars - and countless hours of peaceful sleep. :>))