Runaway RAM when using GPU for Render -- Vegas Error vs GPU/driver

RedRob-CandlelightProdctns wrote on 1/13/2020, 1:41 AM

Since as far back as I can remember, I've run into render issues when using a bunch of tracks with still graphics files on them for composites. Today is no exception, and I decided to spend a few hours running isolated-change tests. Although the problem *could* be pointed to graphics card drivers, since my graphics card is using dedicated RAM, I'm actually thinking there may be a bug in memory handling when data is handed off to graphics cards.

The scenario --

  1. rendering out 28 seconds of a 5-minute promo video. Using Sony AVC/MVC default Internet 1920x1080-30p template
  2. in that 28 seconds, there are about 25 tracks. 17 of those tracks contain transparent PNG files which are layered to create a composite montage. All PNGs are under 3MB, most significantly less; the largest image is 2200x2296. At various points, those graphics are animated.
  3. On 2 layers is the Vegas RAYS effect. One 1 layer is BorisFX's WAVE (it behaves more like a 3d wave than Vegas's native one)

The Problem -- When rendering, as observed via Windows Performance Monitor, memory continues to be consumed and never released, going up, and up and up. Once it hits around 82% Vegas terminates itself. Sarah Connors is nowhere to be found.

Some Tests and Troubleshooting

  1. T1 -- Dynamic RAM Preview = 600, AMD RX470 GPU helping with render -- crashed when Memory rose to 82%
  2. T2 -- Dynamic RAM Preview = 0. Completed render, but memory kept climbing up to 57% at the end. I'm fairly certain if this was a full-project render, memory would be totally consumed and it would crash. Also, saw annoying BorisFX message about "Initializing Mocha" when it crossed a clip which used BCC Remover; I've seen this happen before on occasion and it has slipped into final render. UGH.. wish I knew how to reliably avoid this.

    Here's a sample of the composite region I've been rendering for the tests:

     
  3. T3 -- Same as T2, muted track with BCC Wave. Same results (growing RAM usage)
  4. T4 -- Same as T3, set Dynamic RAM Preview = 200, turned back on BCC Wave. Same results (growing RAM usage)
  5. T5 -- Same as T4, turned off all event FX. No Boris FX, no Rays. Same results (growing RAM usage)
  6. T6 -- Pref->Disabled GPU Rendering, DRAM @ 200 (restarted Vegas). Re-enabled event FX. RAM Stayed Constant at 29-30%! (got that annoying BorisFX Init message again when getting to BCC Remover
  7. I updated my AMD Radeon drivers to the latest version 20.1.1 and configured Radeon to "optimize for compute" (vs "graphics"). Reboot Windows just for shits and giggles. Re-enabled GPU rendering in Vegas (and restarted Vegas).
  8. T7 -- Rendered w/same config as T4 (everything on). Same results. Memory climbed to 57% during render. Performance Monitor shows GPU dedicated RAM only using 1.6 of 4GB, and shared GPU memory @ .1 of 16GB. GPU utilization remains under 7% during render as it has during all the above tests (except for T6 which used 0% GPU)

Based on my tests, it appears that any time the graphics card is used during this render, memory continues to be allocated and not released until it is exhausted. Because it's system RAM that's getting consumed and the video card is using its own dedicated RAM, my inclination is to believe that this is a Vegas memory handling error.

Essential Hardware/Software Config

Because I know y'all will ask:

  • Vegas Pro 17 Build 387
  • Core i7-7700K CPU @ 4.2GHz (4 cores, 8 logical processors)
  • 32 GB RAM
  • GPU -- Radeon RX470, 4 GB dedicated RAM. After updating driver, driver date is 1/9/2020, version 26,20,15011.5007

QUESTIONS

  1. Are there known issues with compositing still graphics that cause runaway RAM with GPU renders?
  2. What is the best way to handle/report/do-further-troubleshooting? Since I've had render problems on multiple computers over time when dealing with multi-track composites, I'd love to see if this can be tracked down and perhaps a long-outstanding bug that could be squashed!

NOTE: I've re-read through Nick H's "FAQ" on rendering issues

Comments

Musicvid wrote on 1/13/2020, 2:48 AM

What happens if you render the same tests without Boris?

RedRob-CandlelightProdctns wrote on 1/13/2020, 4:07 AM

What happens if you render the same tests without Boris?

that was test #5. While I didn't delete the Boris FX, I disabled (unchecked) them on the events which were in the render region.. memory still kept climbing.

not sure it was just a render issue either -- I was having frequent crashes while editing until I disabled the GPU usage.. got through the edit quite smoothly in the last hour.. now time for bed.. 5am EST, and rendering is chugging away pretty smoothly... it *thinks* it'll take 18 minutes more.. BWAHAHAHA.. wait until it gets to that composite region.

RedRob-CandlelightProdctns wrote on 1/13/2020, 2:33 PM

Full project render went smoothly with GPU disabled, and memory remained constant at about 40% through the render.

I know sometimes they say to open a support ticket with Magix, but also know that some Magix devs keep up with these discussions. I've had problems with composites using still graphics for many years, across many versions of Vegas, with both NV and AMD cards.. would be happy to help track this down.

RedRob-CandlelightProdctns wrote on 4/26/2020, 4:22 PM

I had similar problems today, which I've also had for a few years.

When rendering a project with a bunch of photos. In this case, it's just 30 photos, each on their own track, scattered across 3d space using Scattershot. While rendering I see memory use keep going up--- up-- up (as viewed in Task Manager).. once it hits around 80% Vegas crashes. CPU never really went about 35% during the render, and GPU was fairly constant at 11%.

After disabling the GPU and setting Dynamic RAM Preview to 200, the render went without a hitch. Memory remained constant at 26% utilization and CPU running 74-85%.

Runaway RAM when rendering using GPU is a problem. Latest build (build #4) of V17

fifonik wrote on 4/26/2020, 9:20 PM

Can you try to create a demo project?

1. Save As as separate project in separate folder (including media). Check.

2. Remove 3rd party plugins usage in the project (not just disable tracks). Check.

3. Delete everything but the region you rendering. Clean unused media. Check.

4. Replace real medias with generated medias one by one. Check after every step.

5. If the issue is still there -- share (including your VP settings and render options).

People will be able to download and play with it. It is possible someone will be able to find what is causing your issue (It may be png files for example, I never used them in my projects).

Right now there are too many variables so it is hard to suggest anything. I've never seen such memory usage behaviour. My projects are usually quite small, but I'm not restarting PC for weeks and I'd definitely noticed such leaks over time.

P.S. I do not know why in your 32GB system memory usage shown as 70% while VP process is just 4 GB.

Last changed by fifonik on 4/26/2020, 9:27 PM, changed a total of 2 times.

Camcorder: Panasonic X920

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 3700X (not OC), RAM: G'Skill 16 GB DDR4@3200 (not OC), Graphics card: MSI RX580 8GB (factory OC), SSD: Samsung 970 Evo+ NVMe 500MB (OS), HDDs: Seagate & Toshiba 2TB, OS: Windows 10 Pro 1909

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17

RedRob-CandlelightProdctns wrote on 4/26/2020, 9:27 PM

Sure.. I'll do a bit more testing on this one. The only 3rd party plugins I'm using here are for audio, so GPU on/off *shouldn't* make a difference. That said, why not eliminate variables, right? This project was started from scratch so no regions.. full project, about 5 minutes long. I'm pretty certain if I replace the photos with solid media-generator content it will not happen (TBD); I've noted when using still images in the past memory is an issue.

Why are you commenting on 4 GB RAM? Vegas definitely uses more than 4 GB when needed.

 

fifonik wrote on 4/27/2020, 12:25 AM

I'm pretty certain if I replace the photos with solid media-generator content it will not happen (TBD);

Quite easy to test -- create "blank" png with the same resolution or use sample pictures that you can share.

If you find that images causing the issue (VP have to resize the static image at your project fps speed this is why downscaling images is always suggested) you can file re-produceable bug report to Magix.

Camcorder: Panasonic X920

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 3700X (not OC), RAM: G'Skill 16 GB DDR4@3200 (not OC), Graphics card: MSI RX580 8GB (factory OC), SSD: Samsung 970 Evo+ NVMe 500MB (OS), HDDs: Seagate & Toshiba 2TB, OS: Windows 10 Pro 1909

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17

fifonik wrote on 4/27/2020, 12:26 AM

Why are you commenting on 4 GB RAM? Vegas definitely uses more than 4 GB when needed.

On you task manager's screenshot VP uses 4GB. No?

Camcorder: Panasonic X920

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 3700X (not OC), RAM: G'Skill 16 GB DDR4@3200 (not OC), Graphics card: MSI RX580 8GB (factory OC), SSD: Samsung 970 Evo+ NVMe 500MB (OS), HDDs: Seagate & Toshiba 2TB, OS: Windows 10 Pro 1909

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17

RedRob-CandlelightProdctns wrote on 4/27/2020, 1:02 AM

Why are you commenting on 4 GB RAM? Vegas definitely uses more than 4 GB when needed.

On you task manager's screenshot VP uses 4GB. No?

Incorrect. There's different parts of memory reported differently. If you dig deeper than task manager's "memory use" figure and go to the resource monitor, you'l see "Private RAM" (which is the # task manager shows), as well as Shared "Working Memory" ("Modified", and "Standby") RAM. When this problem occurs, it's quite evident that Vegas is eating through all the RAM as Working Memory allocated by the Vegas process climbs until crash-time.

I recorded my tests and am rendering the resulting video right now, which I'll share soon.

RedRob-CandlelightProdctns wrote on 4/27/2020, 1:22 AM

Well, it still might not be reproducible for everyone if it's somehow related to Vegas's integration with the GPU (and if Vegas has different code based on the GPU in the system).. BUT.. I've narrowed down where the bug lies.

My finding is that:

  1. with GPU off, Vegas had no problem rendering JPG photos. Memory was well managed.
  2. with GPU on, Vegas has runaway memory problem when rendering video containing JPG photos, ultimately leading to Vegas crashing when memory was exhausted.
  3. with GPU on and no JPG, replacing the JPGs with text from the media generator, Vegas did *not* have any memory issues!


Conclusion: this appears to be a bug in Vegas's JPG handling.

While scaling down the JPGs may delay the crash for a bit, it does not fix the bug with runaway memory.

I have no problem sharing my .VEG and JPG files with Magix if it will help them troubleshoot. Total size is just 38MB.

The attached 5:25 video link shows the tests and results.

fifonik wrote on 4/27/2020, 2:17 AM

I'd send bug report to Magix (with images).

I tried to put ~100 jpg files (from two mobiles, resolution higher than 4000 x 3000) on time line (two tracks, track motions on each track, simple composition), started to render using SonyAVC to 1080-30p (project props are the same, I tried both 8 and 32-bit pixel formats) and everything was good (memory consumption was just about 500 MB and stable: sometimes goes higher, then lower). GPU enabled (RX580, 8GB, driver ver 20.2.2). Dynamic RAM preview -- 200 (default). 16 threads on my 16-logical CPUs system with 16GB RAM.

So it looks like something in your project (transparent png?) or images (huge exif?) causing the issue.

Last changed by fifonik on 4/27/2020, 11:16 PM, changed a total of 3 times.

Camcorder: Panasonic X920

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 3700X (not OC), RAM: G'Skill 16 GB DDR4@3200 (not OC), Graphics card: MSI RX580 8GB (factory OC), SSD: Samsung 970 Evo+ NVMe 500MB (OS), HDDs: Seagate & Toshiba 2TB, OS: Windows 10 Pro 1909

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17