One "fix" for render 'low on memory'

johnmeyer wrote on 1/18/2012, 11:51 AM
This is for Vegas 10.0e, but I think this may affect Vegas 11.x as well.

I have a very simple project consisting of 1920x1080 60p AVCHD video from my Sony CX700V. Cuts-only (with transitions), plus some audio work. Half hour in total. The only "fX" used was the Mercalli 2.0 plugin (the purchased plugin, not the built-in plugin, version 2.0.96 r77). I documented in another thread that Vegas would not retain the Mercalli stabilization in the rendered version (when rendering to MPEG-2 for DVD), so the workaround suggested was to render individual events to Cineform (which did retain the stabilization) and put these individual Cineform video files on the track above the original event (you can't add them as takes because Mercalli then applies the stabilization to the already-stabilized event, creating the mirror image of the motion in the original clip!).

There are two 1600x1200 still photos at the end, a single MP3 audio track used for opening and closing music, and a title track with about three dozen lower-thirds using the old Vegas text generator. In short: one of the simplest possible projects.

Here's where it gets interesting.

When I tried to render to MPEG-2 using the DVD Architect Widescreen template (modified to 2-pass), Vegas crashed every single time, after about twenty minutes (it should take about ninety minutes to complete). The crash was accompanied by a "low on memory" message, despite the fact that Task Manager showed plenty of memory.

I tried changing the RAM Preview memory, but that made no difference. I tried re-booting and then immediately opening Vegas and rendering. No luck.

I then went back to the project and removed all events to which Mercalli had been applied. Remember, these were on the track below the Cineform clips that I created for each stabilized event so, theoretically, they should not be accessed at all (I left them there so I could still access the original media).

After removing these events, I went to the Project Media list, and clicked the lightning bolt to remove all unused media. I saved the VEG under a new file name/version number.

This time I was able to render without a crash.

So, while this is just one single situation on one individual user's computer, it does seem to confirm what I have suspected, namely that any non-Sony software or plugin must be the first thing to look at when dealing with instabilities or odd situations. It also tends to support the dichotomy that I have witnessed the past few years (since Vegas 10 was first shipped) that some people report no stability problems, while others can't get past first base. While it is possible that some issues may still relate to hardware, I haven't seen much evidence, from those who have reported solving their problems, that this has much to do with anything, other than the very real issues surrounding GPU rendering (my particular problem has nothing to do with GPU, since GPU assist for MPEG-2 rendering does not exist on Vegas 10).

Oh, and for those who want every last detail (system specs, you know), this was rendered on my Polywell custom computer using an i7 Intel processor, under Windows XP Pro SP3 32-bit with 6 GB of RAM (obviously more than can be used by 32-bit Windows) and an nVidia GeForce 9800 GT video card.

Comments

johnmeyer wrote on 1/18/2012, 12:26 PM
I spoke too soon. The resulting MPEG-2 files are full of blank frames.

What utter garbage this program has become. I have 10,000+ posts on this forum, so I I know a little about the program and have read almost every post that others have made in the past ten years. I don't make this statement lightly, or without some knowledge of the situation.

Sigh ...

OK, my next step is to try to get everything into 8.0c so I can finish my project.
Laurence wrote on 1/18/2012, 12:32 PM
>OK, my next step is to try to get everything into 8.0c so I can finish my project.

That or you could just render an intermediate into the now free Cineform GoPro codec.

http://gopro.com/3d-cineform-studio-software-download/
rmack350 wrote on 1/18/2012, 12:35 PM
<edit>Guess I missed the follow-up posts. Bah. Still sucks.</edit>

Probably the only change I'd make is to save as a new file first, then remove tracks and zap unused media. But that's just to prevent myself from accidentally hitting ctrl+s without thinking.

You'd think that video tracks that can't be seen or are muted wouldn't be considered in a render, but that's probably not the case. For one thing, Vegas probably checks that muted track for every frame it renders. A better policy is probably to save as a new file and delete unused tracks and media, as you're illustrating here.

Rob
johnmeyer wrote on 1/18/2012, 12:48 PM
That or you could just render an intermediate into the now free Cineform GoPro codec.Yeah, I think I'll have to do that. I was able to copy/paste a few events from 10.0e to 8.0c, but any attempt to do any more results in a crash of 8.0c. Also, the text events don't copy, and I can't save them via the EDL.

I may first have a go at using the Debugmode frameserver. For some reason, it never installed on my computer for Vegas 10, and I think I know why, but never bother to fix it because I used Vegas 10 so infrequently. I guess I now have incentive to work that out. If I can get V10 to frameserve, I have the external Mainconcept MPEG-2 encoder and I can use that (I hope).
Laurence wrote on 1/18/2012, 1:23 PM
For what it's worth, Cineform is great, even the free version, and since the patch they seemed to have licked the black frame issue. You could also use the Avid codec, but that renders out a lot bigger.
johnmeyer wrote on 1/18/2012, 1:57 PM
Laurence,

Any idea where to get the Cineform patch? I clicked on the link that Dave Newman provided a few weeks ago (in another thread), but the link has expired. Or, do I just download the latest from the GoPro site? I've got 7.5.5.
johnmeyer wrote on 1/18/2012, 3:22 PM
Wow, what a gory, awful mess this project has become because of all the Vegas bugs.

I rendered the project to a Cineform intermediate, but I got the same error (low memory). However, Cineform saves the file as it is rendering, so I have usable video up to the crash point. It turns out it was crashing as it got to one of the two still photos (it's a JPG). So, it is possible that Vegas is still choking on JPEG files.

Anyway, I found a simple cut just prior to this, and am rendering the remainder of the file (including the JPEG) to another Cineform file. I'll then join that with the slightly trimmed version of the ...

... oh wait, it is hanging at the same spot. It definitely does not like this picture. I guess I'll substitute a PNG version and see if it likes that any better.

How can anyone actually get anything done with these later versions of Vegas? Even if you do get work done, you are definitely taking a risky career move because sooner or later, it's gonna get you.
Grazie wrote on 1/18/2012, 3:45 PM
Yup! It's not easy.

G
johnmeyer wrote on 1/19/2012, 10:24 AM
I thought I'd pass on some final details for anyone who stumbles across this thread in the future.

I continued to have problems rendering this project even after I had removed all instances of events with Mercalli. However, it turned out that I hadn't actually removed them all. There was still one that somehow escaped my scrutiny. I found that, rendered it to a Cineform intermediate, and I was then able to render to MPEG-2 from within Vegas.

To get rid of the black frames in the MPEG-2 renders, I downloaded the latest Cineform files (thanks to Dave Newman).

One final problem.

This is a strange one. For each of the several dozen events that I stabilized with Mercalli, I rendered each event to Cineform, ungrouped and then deleted the original video event, and then put the Cineform intermediate in its place. However, the Cineform intermediate was a fraction of a frame shorter in time than the event it replaced. This created a sub-frame gap that could not be seen except at the highest magnification level. I never would have found this except that I had create some software to look for blank frames in the final render (because of the blank frame problem I had in previous renders). It found these blank frames, and I was then able to fix them by clicking on the edge of the Cineform event and making it snap to frame boundaries.

[edit]I forgot to mention: After all this, I still was unable to render from the 32-bit version of Vegas 10.0e, but was able to get a successful render from the 64-bit 10.0e version running under Window 7 64-bit.

Steve Mann wrote on 1/19/2012, 12:41 PM
"The crash was accompanied by a "low on memory" message, despite the fact that Task Manager showed plenty of memory. "

This has got to be the second stupidest error message that the Microsoft Windows engineers came up with. (The first being "No keyboard found. Press any key to continue").

And I have posted on this issue many, many times in the past.

"... but that's not the case because I have X Gb of RAM so that shouldn't be the problem."

Low Memory or Out of Memory does not mean "Not enough RAM", though adding RAM can sometimes fix a "Low Memory" waning. A "Low Memory" warning usually means that you have exceeded your commit limit. You need either a bigger page file, more physical memory, or both.

One of the biggest sources of confusion over Windows memory usage is the whole concept of virtual memory compared to physical memory. The most important thing to realize is that physical memory and the page file added together equal the commit limit, which is the total amount of virtual memory that all processes can reserve and commit.

Start the Task Manager, click on the "Processes" tab, then click on "Commit Size" to sort by size. This will show you which processes are memory hogs that you may be able to shut down.

All Windows since XP (and Unix/Linux for that matter) always wants to have page space. Always. Programs (including drivers and codecs) like to and are allowed to pre-allocate as much memory as they want. Even if they are never ever going to actually use it. Sometimes those programs properly deallocate memory, sometimes they don't (resulting in "memory leak"). Sometimes, programs leave parts of themselves in allocated memory just in case you are going to run that program again. (MS Word, Excel and other Office programs are particularly adept at this). If you have no page file and a program wants to commit some for itself, your PC will crash (AKA, BSOD, or Blue Screen of Death).

Paging file configuration is in the System properties, which you can get to by typing "sysdm.cpl" into the Run dialog, clicking on the Advanced tab, clicking on the Performance Options button, clicking on the Advanced tab, and then clicking on the Change button. I would suggest a value of 1.5X the currently allocated value. The old advice of 2X or 3X your RAM is, well, old advice when a few MB of RAM was normal. The largest paging file that you can select in Windows is 4,095 megabytes (MB).

Also, Windows supports up to 16 paging files, but each must be on a separate volume, so if you have more than one internal disk drive you could try enabling a Paging File on your second hard-disk. DO NOT put a paging file on an external drive because if it's not present when Windows boots, then Windows will crash.

For more information, see http://support.microsoft.com/kb/237740 and http://support.microsoft.com/kb/2267427

Let me repeat for emphasis: The largest paging file that you can select in Windows is 4,095 megabytes (MB).



johnmeyer wrote on 1/19/2012, 2:19 PM
Useful information, although speaking strictly for myself, I already knew most of this and have paging files on multiple volumes.

One extra insight is that you should not completely eliminate the paging file on the C: drive. Windows doesn't crash, but it isn't exactly happy either. I keep C: drive exceptionally small (10 GB) in order to simplify image backups. Thus, I don't want to keep a large paging file on this volume. However, when I tried setting it to zero, bad things happened. During my research on what was going on, I found out that it should not be set to zero.

Of course any time the paging file is actually used, you get into a "thrashing" situation, and everything slows down to about 1/100 the normal performance.

Finally, back to the original problem, it is not at all clear to me that, despite your excellent advice on paging files, that this particular "low on memory" error message is actually generated by Windows. It certainly is not a Windows dialog box that appears, although Vegas could obviously be getting a system message and then passing that on using its own alert.

My own sense of what is going on is that this is a memory problem specific to Vegas and, like the RAM Preview setting that interacts with all sorts of other things, there is probably a solution that will have to come from the Sony engineers, and therefore this cannot be cured or avoided simply by playing with paging files or RAM. There is certainly no sense of thrashing or slowdown that accompanies the problem.
john_dennis wrote on 1/19/2012, 6:56 PM
Even though I have placed the pagefile on drives other than the boot disk for some time, I read somewhere a long time ago that when Windows dumps, it will only write to the boot disk. Because of that, I have kept the pagefile on the boot disk larger than my miserly ways would have demanded. I couldn't tell you exactly how big it is because of my poor personal memory and the fact that I don't set the system up but every four or five years. I just reimage back to a base install as long as the hardware stays essentially the same.

500 KB or 1 MB comes to mind but, then, my personal memory has no ECC.

Edit after wet drive home.

With the arrival of the SSD. I put all pagefile on the boot drive. If it wears out, I'll buy another one.
R0cky wrote on 1/19/2012, 8:48 PM
Steve, 4095 MB is not the limit. Mine is set to 28 GBytes which is an entire partition I have set up just for the page file. I have 24 GBytes of ram.

rocky
johnmeyer wrote on 1/19/2012, 9:07 PM
Steve, 4095 MB is not the limit. Mine is set to 28 GBytes which is an entire partition I have set up just for the page file. I have 24 GBytes of ram.I think what Steve said is essentially correct, although you can certainly have more in aggregate. See the following Microsoft support article:

How to overcome the 4,095 MB paging file size limit in Windows
Steve Mann wrote on 1/19/2012, 9:39 PM
John, that article is specific to Windows XP and Windows Servers.

Rocky, you can set the pagefile to any size you want, but Windows only uses 4Gb of it.
Rory Cooper wrote on 1/20/2012, 1:12 AM
John I find clearing out unused media in project media as you have done helps with processing speed especially when working with nested veg files.

I installed Matroska splitter and ffdshow and klite so don’t have issues.

How are you rendering out Cineform from Mercalii? Prodad fx has a mcsv codec for Sony Vegas work around so I presume this is the default for Mercalli.

Question …if I create a subclip after mercalli ap is Vegas treating this as a new separate piece of media if so this might help without all the pre-renders going on
LoTN wrote on 1/20/2012, 4:44 AM
Rocky, you can set the pagefile to any size you want, but Windows only uses 4Gb of it.

By Mark Russinovich himself:

"A couple of final limits related to virtual memory are the maximum size and number of paging files supported by Windows. 32-bit Windows has a maximum paging file size of 16TB (4GB if you for some reason run in non-PAE mode) and 64-bit Windows can having paging files that are up to 16TB in size on x64 and 32TB on IA64. For all versions, Windows supports up to 16 paging files, where each must be on a separate volume."


Pushing the Limits of Windows: Virtual Memory
riredale wrote on 1/20/2012, 11:18 AM
Guys, I am mystified by much of this.

I'm a dinosaur, perfectly happy to continue running Vegas7d with my HDV projects (almost every job I do is with two HDV cameras and multiple audio sources). I'm also stuck in the past by running XPpro32.

For many years I have had a little freeware utility called "RamPage" in my system tray. Rampage simply shows real-time how much ram is currently idle. Several years ago I boosted my ram to a then-breathtaking 2GB (!) and noticed that RamPage showed 1GB to be lying fallow most of the time. I was also carrying 2GB of paging memory. As an experiment I eliminated the page file completely, relying on just ram alone. Again, RamPage showed 1GB of ram just hanging around doing absolutely nothing most of the time.

I found that an intensive render in Vegas7d could eat up 200-300MB of extra ram, as could a second instance of Vegas or a DeShaker render. But I had a hard time using up ALL of the 2GB of ram. If I did, the system would predictably hang.

So the conclusion I drew was that, at least under XPpro and V7d, memory was never going to be a constraint. Perhaps under Vista and now W7 and the latest versions of Vegas that isn't the case any more, but that was how it was just a few short years ago.

Oh, and as I've mentioned before on this forum, V7d is as stable as a rock. Never any issues. Can't do AVCHD, but I assume I could use V8 for that if necessary.

As I'm typing this RamPage shows 1042MB of ram doing nothing. Task Manager shows 82 processes running in the background. System runs 24/7 (it also acts as a web server).
Steve Mann wrote on 1/20/2012, 9:24 PM
Ram use has little to do with pagefiles. Except that your physical memory (RAM) plus the pagefile is the total commit limit - the memory resources available to any program.
Steve Mann wrote on 1/20/2012, 9:32 PM
"By Mark Russinovich himself:"

Mark also says that Microsoft often gives conflicting information. My referenced Microsoft support article is older than Mark's latest post on this subject, so for now, I'll remove the 4Gb limits from my boilerplate response.
johnmeyer wrote on 1/21/2012, 12:10 AM
Oh, and as I've mentioned before on this forum, V7d is as stable as a rock. Never any issues. Can't do AVCHD, but I assume I could use V8 for that if necessary.Yes, that's what I'll do in the future. I have, for my use, confirmed beyond any shadow of a doubt that Vegas 10.0e is unusable and unstable. I only shared a few of the dozens and dozens of problems I encountered (like psychadelic MPEG-2 renders, blank frames even without Cineform on the timeline, crashes even without any plugins, etc.).

So, in the future, all my editing will be done in 7.0d or 8.0c. I'll convert to Cineform and edit with that. At this point, the only thing that is really useful (for me) in all the stuff that's been added since 8.0c is the Elastique and the ability to put MP4 directly on the timeline.

The real test of whether an upgrade is worth the headaches is whether you feel cheated or lost when you have to go back to the earlier version. In my case, I actually feel a great sense of relief and a return of confidence when I get back to 7.0d or 8.0c. This is even truer, if that is possible, because the UI changes in the later versions are, without any exception, a step backwards, or a pointless sideways step at best.

I've kept silent on this for some time because I really want to support Sony. However, I killed almost a week of my life chasing down all these bugs and problems, and did so on a project that was personally really important to me (my daughter's run in the Houston Olympic Marathon Trials). It is one thing when you have a crash and lose an hour's work; it is completely another when you lose several days of time, and lost that time constantly -- every hour -- always having to find some way around a brand new problem.

And, just to make a point for all those people who love to say, "it works on my computer, so it must be something in your hardware or software."

No, the problem is that Vegas 10.0e is buggy.



Grazie wrote on 1/21/2012, 12:33 AM
John, will you even consider dabbling in VP11?

G

johnmeyer wrote on 1/21/2012, 12:42 AM
I briefly tried V11 and found absolutely nothing in it that was useful for me because, when it was initially introduced, it did not support my GeForce 9800GT video card. As far as I can tell, the major, "meaty" feature in V11 is the promise of much faster rendering speed (which, if it works, is indeed a major feature).

When support for 9800GT was added in a later release, I was going to try it out, but my trial had expired, and when I contacted Sony Customer Support and explained what happened and asked if they could give me an extension to the trial, they said that they never, ever do that. Fair enough, but that means that I cannot possibly tell whether it works without forking over quite a bit of money and, since Sony Creative Software has forfeited my trust, I'm not going to spend any more money.

I'm not sure, Grazie, why you have persisted with V11. I've read about your various problems, and I'm not sure what goodies have enticed you to stay. If you tell me that it renders faster, fair enough, but if that's the only thing, then the smart and logical thing is to edit on a stable platform (V7, V8, or maybe V9), and then open the final project in V11 and use it simply as a rendering engine.

However, I've had so many problems with V10 that I actually wrote software to look for single blank frames in the rendered file. Of course, just when I thought I had that problem licked, I ended up with totally corrupted video with back and forth motion, a little like field reversal, but on steroids. I found the cause of that corruption (another bug in how Vegas works with Mercalli), but then I ended up with a half hour MPEG-2 file that was totally psychedelic (inverted colors). I had read about all of these problems in this forum over the past two years, but somehow I managed to get all of them to happen to me in the space of a week.

Meantime, I finished that project with 15+ hours of video on the timeline, and over 14,000 separate events, and not one single crash in Vegas 7.0d.

I'd better stop. Like I said in the post above, there are a lot of bugs and problems that I haven't shared yet, and that's yet another one.

Grazie wrote on 1/21/2012, 3:09 AM
John, thanks for your open reply to my question.

I'm gonna ask what others may/could be thinking - why not put yourself forward as a BETA tester? Your breadth of knowledge on IT and your background in Video has got to be a real boon for SCS.

Why was I enticed to try VP11? I wanted access to the OFX platform that 3rd Party providers could make use of and develop in VP11. And yes, the idea and outcome of using dormant RAM - viz the GPU - appears, on the face of it, a go-to option. All of my quest has been to get to full fps, in an adequate enough fashion for making creative decisions and for the system to remain stable. Presently I did hit a mix of RAM for rendering that put all my PC in a spin and I got blistering rendering at MP4 and MPEG2. And yes, you've nailed it: I edit in VP10e, it's quite stable now, but I'm queasy, and have been rendering out in VP11.

I don't do volume Media, as do you. What I do do is PUSH Vegas real hard and very fast to get to a creative decision, using Tracks+OFXing+MP4 rendering out to clients through DropBox. I came to Vegas being blown away just how EASY it was to get to that all important creative decision could be achieved. That was my particular tipping-point then, and this still remains my Holy Grail to this day.

Just a point of information: I HAD to upgrade my PC - I actually started all over again with a new box - and had my builders work on something that would drive me towards stability. You may have already read my specs, it is System 2.

I'd still like to think that you'd consider and then approach SCS to "assist" them.

Cheers

G