Vegas Pro 8b Render Problem Identified

Darren Powell wrote on 2/7/2008, 3:27 PM
Hi everybody,

Firstly, thanks to everyone who has been responding and helping with my render (and exception error) problems in Pro 8.

There is no problem with the computer - plenty of RAM, plenty of HD space, plenty of cores etc. etc.

I spent the better part of the day yesterday with Mark Harwood from Sigmacom / New Magic here in Sydney Australia. Mark's great, highly recommend him for any of your purchases / tech support etc. if you're based on this side of the planet.

We deconstructed my project and, to cut straight to the point, we identified that Vegas Pro 8b has an upper limit on virtual memory usage.

My feature film is about 90 minutes long with up to 220 seperate video and audio tracks. Some of you might say 'hey that's excessive' ... I don't agree ... I've made multiple 47 minute doco's in previous versions of Vegas on very slow computer's with less RAM than I have on my video card right now ... and they were fine ... I expect Vegas to keep up. As a matter of fact I'm sure that if I'd kept my project in Vegas 7 I wouldn't be having these problems.

(As an aside, not all of my audio tracks are loaded up with tons of wav files ... most of the tracks only have one or two events that happen at various points in the film ... sound FX etc... I just like to have individually named tracks ... and total control over every event in 5.1 surround space for my final mix ... it just makes everything clearer as I mix. Anyway, the number of tracks I have is not the problem ... )

The point is: We have identified on two different quads sitting side-by-side just yesterday ... that Vegas Pro 8b has an upper limit when it comes to virtual memory. As soon as my project exceeded 1.2GB's of virtual memory ... the project was 'unrenderable'... ie: on both machines Vegas locked up on frame 0 and would not render. Full Stop.

When Mark and I laboriously cut and pasted the first 10 minutes of the project (Multiple Video and up to 50 audio tracks at a time) into a new Vegas project we were able to render with no problems until we loaded the virtual memory up to 1.2GB and then everything stopped.

The really stupid thing is that the computers each still had a couple of Gigs of onboard RAM which Vegas didn't seem to be interested in at all!!

Mark Harwood has been in touch last night with the Sony Software guys to address this issue ... I think it may be as simple as writing a patch allowing Vegas to use real memory and not just the page file!

Who knows ... but at least - for the moment - I have a (very long winded and time consuming) work-around for the film.

Anyway, that's the latest ... thanks again everyone ... really appreciate your help and I hope this post helps people who are planning on building a decent sized project in Vegas Pro 8b. Don't let your project segments get to 1.2GB of PF Usage as reported in Task Manager ... and you'll be fine! :-)

Cheers,

Darren Powell
Sydney Australia

Comments

rmack350 wrote on 2/7/2008, 4:28 PM
Thanks Darren, it's good to know. Also good to hear that you found a vendor who can get SCS to answer the phone.

We've had similar go-rounds with Adobe and Matrox and our system integrator has been invaluable in getting us in contact with people inside the companies.

Rob Mack
Xander wrote on 2/7/2008, 5:11 PM
Interesting. Wonder if this explains my 32bit rendering issues. I get the same symptons. I'm pretty sure the memory usage must go up 4x in 32bit mode. Will check the page file usage when I get home. Anybody know if Vegas 7 has this memory issue?
rmack350 wrote on 2/7/2008, 5:25 PM
You know, if you think about it, all programs built for Win32 have a 2.0 GB limit. That is a combined RAM+Pagefile limit. It seems to me that Vegas was just plainly running out of memory.

What's not good is that Vegas doesn't deal with this gracefully, and THAT is a big problem for users.

The limitation doesn't go away if you run Vegas on Win64. It's still a 32-bit application built for a 32-bit OS. It's still built to use only 2GB, which is normal for such an application.

Remedies that SCS could pursue?
--Better memory managment/memory usage (find ways to release things from memory)
--Better warnings.
--Write and compile to use 3GB. This is a bandaid, but it would allow a 32-bit application to use up to 3GB if you run it under win64. PPro is built this way (and PPro needs it more)

Rob Mack
Darren Powell wrote on 2/7/2008, 5:54 PM
Yes, 32bit colour space is impossible to use because of this memory limitation at the moment ... that's seems pretty obvious now ... but it's still only a theory.

Yes Rob, better memory management, write and compile to use as much memory as is available ... etc etc ... when Vegas is ported to 64 bit I would expect Vegas to be able to access 8 or more Gig ... plus PF ... why not? Otherwise this whole business of finally arriving at 64 bit computing seems pointless.

I said to Mark yesterday (tongue in cheek) ... the problem still dates back to when Bill gave us 640KB of memory (no-one will ever need any more than this - come on guys I've just given you 10 times more than my original upper limit of 64K!!) ...

Of course we wanted more so Bill wrote (under much duress) the EMM ... which addressed the 'add-on' to that Holy Grail of memory from 640KB to 1MB of RAM ... !!

I have an idea ... why don't we just DO AWAY with upper limits on memory and write software that's free to address ALL available RAM whether under a 32bit OS or 64bit OS ... and if it runs out of RAM ... then use the pagefile on a harddrive to get the system out of trouble!!

There's an idea for you! Now I have to get back to cutting and pasting my 'massive' project into tiny 10 minute chunks so that Vegas doesn't get indigestion.

Cheers and appreciate the comments.

Darren




Laurence wrote on 2/7/2008, 7:19 PM
The 64 bit version of Vegas that was announced last year would fix this nicely for those of us using Vista 64.
Terje wrote on 2/7/2008, 7:22 PM
the problem still dates back to when Bill gave us 640KB...

This isn't correct. Well, unless you are using Windows 95 or 98 that is, and I seriously hope you are not. There is no relationship whatsoever between the memory limits of Windows XP/Vista/Mac/Unix and the artificial memory limits imposed upon the world by the 8088/8086 CPU and therefore DOS. Do note that this wasn't really a Microsoft problem, it was an Intel/IBM problem. DOS just stayed alive a lot longer than anyone anticipated and carried the issue into the 80386 world where it didn't really belong.

why don't we just DO AWAY with upper limits on memory

That isn't all that simple. Not unless you want really slow computers. In theory a memory address could have an arbitrary amount of bits, it would have to have an end marker or something to identify the end of the address, but that isn't that much of an issue. The issue would be that in order for the CPU to address something it would have to do a hell of a lot of computations. For each memory access. That would be sloooooow. Very slow.

Instead computers have registers it can use for memory access. These have a fixed size, and moving things from memory and onto the CPU dye for processing then becomes quite efficient. Also, for the CPU to alter things in memory becomes fast and efficient. Those are the two things you want to be fast and efficient.

A register has a fixed size. In most CPUs today it is typically either 32 bit or 64 bit. It has been 8 bits, 6 bits, even 4 bits, 12 bits and 24 bits. If you have a 32 bit register the maximum amount of memory a computer can access becomes 4G. Now, then we have the operating system. That has to deal with all kinds of issues besides managing memory. It therefore usually sets aside a certain portion of the address space (not the actual memory) for it's own purpose. In XP and a number of other OSs, that is about one third to half of the address space.

So... no, we can not do away with upper limits on memory. We can make them very large, but then again, every time someone makes something that can fit a lot of data, someone comes along figuring out ways to create a cr@pload of data.
Darren Powell wrote on 2/7/2008, 7:35 PM
Thanks for clarifying that for me Terje ... ! I did actually write 'tongue in cheek' ... which is a warning bell indicating that the following comment is designed to be a joke based on the fact that I've just been in computer hell for 2 months working out that Vegas Pro 8 freezes up if your individual project is indicating more than 1.2GB PF usage!!

ie: I'm only asking SONY to get Vegas working with memory better!

And yes, as long as the systems get bigger and the machines get faster I'll be one of the people throwing cr@ploads of data at my projects ...

I mean for goodness sake ... what's the Vegas Pro 8 advertising slogan ... 'Imagine the impossible'??

Damn I should have taken a closer look at that sales pitch! If I was only supposed to 'Imagine the Impossible' I would have stuck with Vegas 7. DAMN!

Darren Powell
Sydney Australia
rmack350 wrote on 2/7/2008, 7:47 PM
Well, a 64-bit version should be able to access tons of memory, and it looks like Vista 64 would let it, so your limitation will start to be in chipsets, CPUs, and motherboards. Since Vegas is just now crapping out for you, I'd think you wouldn't need much more memory, and if it'd been compiled to be Large Address Space Aware you'd probably be happily working right now.

Given that SCS announced they were working on a 64-bit Vegas, I assume they were seeing this coming. Given your description of your project, this is something we could never even approach in our shop using ppro.

Rob
rmack350 wrote on 2/7/2008, 7:53 PM
I guess you'd rather imagine the possible ;-)

Rob
Darren Powell wrote on 2/7/2008, 8:03 PM
Thanks Rob,

Yeh, the project isn't really that big when you break it down ... it's only a feature film ... Robert Rodriguez has been doing this kind of stuff on an AVID for years ... Vegas just needs to use memory more efficiently and we'll be screaming along.

Cheers again,

Darren
busterkeaton wrote on 2/7/2008, 8:18 PM
I wonder if this memory thing is only happening with Quads.

Darren, for an immediate solution to your problem, have you tried using nested projects?
Darren Powell wrote on 2/7/2008, 8:22 PM
Hi busterkeaton,

Yes, I was initially carving up big chunks of the project and then nesting those .veg files into a 'Final Version' project. The problem was that because the project is so dynamic I'd have to be constantly re-rendering quite large nested files before I could continue working on the 'Final Version'... and then the machine would fall over because the 'Final Version' project was obviously hitting the PF upper limit ...

Anyway, I think my 10 minutes max per project solution is becoming my final work-around.

Thanks for the post.

Darren Powell
Sydney Australia

PS. Regarding Quads ... I did have marginally greater success rendering from my Dual Xeon box sitting right next to the Quad ... but still had lock up problems at the end of the day ...
PeterWright wrote on 2/7/2008, 9:04 PM
Hi Darren

It's probably too late now, but when you talk of laboriously cutting and pasting 10 min multi track segments into new projects, wouldn't it be quicker to Save as, then use the Selection Edit tool to select and delete all but the 10 mins you want to keep?

If that's what you're already doing please ignore this ...
UlfLaursen wrote on 2/7/2008, 9:25 PM
Good news Darren - it sure seems to have been a long walk for you, but it seems to have paid off. Glad we have pioneers like you and your mate. Hope that Sony will get this fixed in the next update.

/Ulf
Terje wrote on 2/7/2008, 10:03 PM
Yes, 32bit colour space is impossible to use because of this memory limitation at the moment

I don't think this is related to this memory issue. Vegas goes belly-up on me with very small HDV projects in 32bit color space. There is no way I am running into memory allocation issues with those small projects. It may be a memory issue, but it is definitely not related to allocating 2G of memory.

Still surprised this has not yet been adressed by Sony.
rmack350 wrote on 2/7/2008, 10:19 PM
Yeah. Robert Rodriguez.

I'd guess that his edit projects were segmented a bit. You know, send the audio out to an audio app, do the compositing outside of the edit system. Vegas kind of lulls you into doing more in one project so I can see how it'd be easy (and dangerous) to build up a very big project. There's nothing about Vegas that prompts you to break down a project into smaller parts. To put it another way, Vegas gives you enough rope to hang yourself.

Rob
Darren Powell wrote on 2/8/2008, 4:42 AM
Thanks Laurence, Peter, Ulf, Busterkeaton, Terje and of course Rob for your comments, support, ideas etc. .. (and anyone I've missed...)

Terje, I agree the 32bit colour space thing probably has issues beyond just a memory problem ... as you say, I hope Sony address it soon because things start to look really nice in 32bit ... (when it's working briefly ...)

Peter, that was the point of the 'cut and paste' ... as soon as you open the 'big project' in Vegas (as Rob said, the one you've effectively hung yourself with ... by building it so quickly and easily in Vegas) ... you can't just select and delete stuff because it appears that 'the whole project' remains open in memory ... no matter how much stuff is (or isn't) on the timeline ... that's what was happening yesterday on two quads anyways ... so with the whole project open in memory and even with a small cut on the timeline ... I couldn't render ... simple as that ... (thanks for posting though!)

Therefore, we had to cut and paste into a 'new project' until we approached the dreaded 'gallows' in memory ... which was consistently about 1.2GB on two different machines with 4GB of RAM ... and all the other bells and whistles... as soon as the PF usage indicated 1.2GB (with nothing else much open in Windows XP) ... Vegas simply didn't render anymore.

But now, as of a few minutes ago ... it's now 11.28pm in Sydney Town ... I have just rendered the first 10 minutes of my movie as a .wmv (not bad res ... 35MB in size) to send to a distribution company in Germany.

Wow, who would have thought it would take 6 weeks and no sleep to render 10 minutes as a .wmv!!

Anyway ... as Frank once said ... 'That's Life'.

(Shameless Plug ... If you want to hear my versions of some of Frank's songs - head over to www.darrenpowell.com - for a bit of a break if you like!)

Thanks again,

:-)

Ciao for now.

Darren Powell
Sydney Australia

farss wrote on 2/8/2008, 6:32 AM
A couple of questions spring to mind.
How / why is V8 using so much RAM to load a project?

I'm pushing the proverbial up a steep hill to get a project file to exceed 1MB, even with 220 tracks we're still talking MBs, not over 1GB. The thing is of course Vegas also likes to build waveform files and thumbnails. Assuming it loads all the thumbnails and waveforms into RAM then we start chewing up RAM at a great rate of knots.
Two simple suggestions spring to mind. Build 8 bit Waveform Files would save a fair slab of RAM. Turning off Thumbnails and Waveforms should really save even more RAM.
More complex suggestions that'd require code changes. Only build thumbnails for the first and last frame of a clip like PPro does, now that's got to save buckeloads of RAM. Make separate switches for enabling thumbnails and waveforms.


Now the nasty bits :(

It's great that Mark devoted the time to get to the bottom of what was causing the problem. I admire your perserverance, I dread to think how things would have panned out if you had overheads to pay while you spun your wheels on a commercial project, the kind where someone's underwiting your movie to the tune of a couple of million and the bankers are charging interest while you wait to get a return from the box office.

But hang on, didn't the guys that wrote this code know about this limitation, didn't you buy an NLE with "Unlimited tracks"? Did they figure no one would actually try to do a Pro job on their "Pro" NLE, or did they figure only a few would and they'd get so pissed off they would go elsewhere and what's a few users dollars anyway.

And what happened when you raised a trouble ticket, did the issue get properly escalated to the guys who knew the answer all along, nope.

The final irony. The last PPro seminar I went to the second thing their man said was "we know we've got some memory issues, if you're doing a longform complex project we highly recommend you split it up into reels". Makes for a pleasant change, that honesty thing does.

Bob.
CorTed wrote on 2/8/2008, 8:46 AM
Darren, I must commend you on your persistence in trying to find the source of this problem. I hope this will get SCS to take a serious look at this problem. I have been posting serious memory problems with VP8 since November of last year and noting that the program becomes un-reliable after using approx 60-70% of the available memory. I have told support about this problem since 8a

----------"The 64 bit version of Vegas that was announced last year would fix this nicely for those of us using Vista 64."-----------

That is all fine and dandy for the 64 bit users, how about fixing it for the 32 bit users now!

I truly hope SCS will take a real good look at their memory problems outlined in these posts and get a fix out ASAP.
(Although I'm beginning to wonder if they really look/listen.... maybe they do not know how to fix the problem.........)

Ted
Darren Powell wrote on 2/8/2008, 2:56 PM
Thanks Bob and Ted,

That's exactly how I feel ... I remember years ago trying to build a mulitmedia project in some software called 'IconAuthor'. - it was the price of a small car at the time not to mention being charged for updates. On the box it said 'IconAuthor' is the most comprehensive 'cross-platform' authoring tool on the market.

Guess what ... I took the bait ... (think I would have learned by now) ... built a project on a PC ... multi-level CD-ROM with a shopping mall and shops and sounds and products ... you know ... the sort of thing that was all the rage at the time ...

Came time to deliver to the client ... I'm looking in the box for the 'Mac Player' as advertised ... guess what ... there was no Mac Player! Got onto the people at IconAuthor (again in the US so I'm half a day out of whack!) ... they tell me on the phone at 2.00am ... 'Sorry the Mac Player is in Beta Release at the moment' ... I'm screaming down the phone WHAT!!!

Anyway, they send me a copy of the Mac Player (for their product which is the best 'cross platform' blah blah blah)... powered up a Mac and loaded the ROM ... all of the videos are clumped in the middle of the screen instead of nicely arranged around the place ... all the graphics are clumped one on top of the other in the middle of the screen ...

IconAuthor is no more of course ... but I sense that both of your comments guys are pretty accurate ... I'm just hoping (and praying because I need to get this project finished soon) ... that Sony will make it a little bit easier to render this stuff asap!

Cheers, back to my 10 minute reels ... and I'll try switching off those thumbs ... if I can find the right switches!

Cheers,

Darren
Laurence wrote on 2/8/2008, 3:05 PM
Darren, am I correct in assuming your project is HDV native rather than Cineform codec? It seems to me that in order to be able to cut long GOP at any given frame, Vegas must be buffering a whole lot of surrounding frames at each start and stop point in order to be able to do this. You could probably go quite a lot further with a Cineform project before you'd run into these memory problems.

None-the-less, I agree that it's something that needs to be fixed, even if it's not an easy fix.
rmack350 wrote on 2/8/2008, 3:41 PM
I think the thumbs can be switched off with ctrl+Shift+W.

The other suggestion for breaking down the project would have been like this.

-Open whole project
-Set a marker in the middle of the project and save
-Save again as "My Project Part1"
-Delete last half after marker and save.
-Open original project again
-Save as My Project Part2
-Delete first half of project up to marker.

Just another way to get the project broken up rather than Copy/Paste.

Other ideas? Model it in a more traditional workflow. Make a copy of the project for audio and one for video. Assuming that picture is locked, remove all the audio from that video project and try to render to a decent intermediate format. If you can do that then put the redered video into the audio project and delete all the other video tracks. Just edit the audio in this project.

I think this probably simulates what you'd do if you had to send the project out to a Protools system.

'Course maybe you're already doing this.

Rob
Terje wrote on 2/8/2008, 11:36 PM
The 64 bit version of Vegas that was announced last year would fix this nicely for those of us using Vista 64.

That is unlikely if they use much of the same code base as Vegas 32 bit (recompiled for 64 bits). There is no reason that Vegas, if properly coded, should have these problems to the degree it does. So, there are some serious problems in the Vegas code. Those problems don't go away magically by going 64 bit.

When I run Vegas on the projects that die, it never goes above 1G of memory usage. This isn't a 32bit vs 64bit issue, this is an issue Sony engineers need to fix in the current version of their code. If they don't it will probably also break in the 64 bit version.
Udi wrote on 2/9/2008, 2:31 AM
Before saving the project, remove all unused media from the project. Otherwise it is still opened and consume memory.

Udi