Rendering freezes with RTX 5080 card and Nvidia drivers over v572.83

BrettW wrote on 8/16/2025, 7:41 AM

I must say, this is quite a frustrating matter that has persisted since February of this year, specifically concerning AC-3 6-channel audio, even with the current Vegas Pro 22 build 250.

When rendering 1080i25 AVC videos with AC-3 6-channel audio to 1080p50 using the MAGIX AVC NVENC encoder and 2-channel audio, VEGAS freezes. The rendering progress stalls, the preview window becomes unresponsive, while the graphics card remains hard at work, noted by the speed of it's fans and confirmed by checking the Performance tab in Task Manager. The rendering process fails to complete, freezing at random places on different tries, necessitating termination via the Task Manager on every occasion. It only happens while AC-3 6-channel audio is involved, so I'm wondering does MAGIX's AC-3 decoder need a tweaked update to work correctly with the later Nvidia drivers for hardware rendering while using an RTX5080.

The only thing that works flawlessly, is if I downgrade the Nvidia drivers back to 572.83. Those drivers were released way back in February this year. Using any later versions of Nvidia drivers crash the rendering process as I explained above. (Additionally, the necessary later Nvidia drivers are required for various modern games.)
Furthermore, switching the "GPU acceleration of video processing" setting from the Nvidia GeForce RTX 5080 to Intel Graphics provides a workaround, albeit at the expense of NVENC encoding performance benefits.

I have many of these interlaced m2ts files I wish to edit and convert, hence I'm looking for a fix.

The following is simply the MediaInfo data for one of my original 1080i25 m2ts files with 6-channel audio, which is intended to be rendered to 1080p50 with a 2-channel audio configuration.

General
ID : 1 (0x1)
Complete name : E:\Digital Video\Vegas\Home Videos\2007-10-13.m2ts
Format : BDAV
Format/Info : Blu-ray Video
File size : 2.64 GiB
Duration : 22 min 40 s
Overall bit rate mode : Variable
Overall bit rate : 16.7 Mb/s
Maximum Overall bit rate : 35.5 Mb/s
Frame rate : 25.000 FPS

Video
ID : 4113 (0x1011)
Menu ID : 1 (0x1)
Format : AVC
Format/Info : Advanced Video Codec
Format profile : High@L4
Format settings : CABAC / 2 Ref Frames
Format settings, CABAC : Yes
Format settings, Reference frames : 2 frames
Codec ID : 27
Duration : 22 min 40 s
Bit rate mode : Variable
Bit rate : 15.6 Mb/s
Maximum bit rate : 16.0 Mb/s
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate : 25.000 FPS
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Interlaced
Scan type, store method : Separated fields
Scan order : Top Field First
Bits/(Pixel*Frame) : 0.300
Stream size : 2.47 GiB (93%)

Audio
ID : 4352 (0x1100)
Menu ID : 1 (0x1)
Format : AC-3
Format/Info : Audio Coding 3
Commercial name : Dolby Digital
Codec ID : 129
Duration : 22 min 40 s
Bit rate mode : Constant
Bit rate : 448 kb/s
Channel(s) : 6 channels
Channel layout : L R C LFE Ls Rs
Sampling rate : 48.0 kHz
Frame rate : 31.250 FPS (1536 SPF)
Compression mode : Lossy
Delay relative to video : 25 ms
Stream size : 72.7 MiB (3%)
Language : English
Service kind : Complete Main
Dialog Normalization : -31 dB
compr : -0.28 dB
cmixlev : -3.0 dB
surmixlev : -3 dB
dialnorm_Average : -31 dB
dialnorm_Minimum : -31 dB
dialnorm_Maximum : -31 dB

Comments

RogerS wrote on 8/16/2025, 8:06 AM

Furthermore, switching the "GPU acceleration of video processing" setting from the Nvidia GeForce RTX 5080 to Intel Graphics provides a workaround, albeit at the expense of NVENC encoding performance benefits.

I don't understand this. NVENC is not directly connected to the GPU you select in preferences/ video. You could have an Intel iGPU power your timeline (not that I would) and then the 5080 do the encode. You can pick either GPU for decoding in preferences/ file io. So it's mix and match for decoding, encoding and timeline/Fx processing.

Is there any error message in VEGAS when it crashes? Anything in Windows reliability history? Maybe you could ask do a memory dump from windows task manager when VEGAS crashes and send it to support to analyze.

Howard-Vigorita wrote on 8/16/2025, 11:44 AM

Gpu selection is not necessarily under the total control of the Vegas application. The Nvidia Control panel has a link to the Windows graphics settings that are now supposed to be in control. There are also Nvidia Control Panel's own settings for global behavior or for the Vegas application specifically. I used to have to fiddle with the Windows application-specific gpu assignments, but have not needed to do that since vp21. And I haven't had to mess with Nvidia application-specific settings since Microsoft wrested control away from it. But Nvidia may have come up with a way around Windows and application control for the 5000-series gpus... playing with those settings might help.

RogerS wrote on 8/16/2025, 9:28 PM

I think as long as both GPUs show up in VEGAS there's no need to change Windows settings (at least here GPU activity shows up in the expected place when switching values in VEGAS). The NVIDIA app seems to be supplanted by Windows Graphics.

BrettW wrote on 8/17/2025, 8:43 AM

Furthermore, switching the "GPU acceleration of video processing" setting from the Nvidia GeForce RTX 5080 to Intel Graphics provides a workaround, albeit at the expense of NVENC encoding performance benefits.

I don't understand this. NVENC is not directly connected to the GPU you select in preferences/ video. You could have an Intel iGPU power your timeline (not that I would) and then the 5080 do the encode. You can pick either GPU for decoding in preferences/ file io. So it's mix and match for decoding, encoding and timeline/Fx processing.

Is there any error message in VEGAS when it crashes? Anything in Windows reliability history? Maybe you could ask do a memory dump from windows task manager when VEGAS crashes and send it to support to analyze.

That's correct Roger. NVENC is only fully achievable if the NVidia GPU is selected in preferences/ video, as this GPU acceleration setting handles rendering as well as preview playback and appyling effects. But this is strangely causing the lockups during rendering with my scenario.
Hence, switching it to Intel Graphics just prior to rendering, allows it to fully render, albeit drastically slower. And having Intel Graphics selected even while still using an NVENC template, the outcome is that it doesn't lock up and the rendering time is still quicker than using an Intel QVS template. Strange, but true. As I mentioned, it's a workaround, not a solution.
And yes I'm aware decoding is handled in preferences/ file io. I always leave that set to Auto (Nvidia GPU), as I don't have issues working in the timeline. And it's quicker.
I've even tried changing hardware decoder in preferences / file io to use the iGPU and GPU acceleration in preferences/ video to use the GPU, but rendering still locks up.

I do need to make a correction in my previous post. When it locks up during rendering, I mentioned that Vegas continues to use additional PC resources, That's true, but it is not GPU resources, but only CPU resources, generally around 20%. GPU resources drops to 0/1%. My mistake.

I don't receive any error messages in Vegas, nor is there anything showing up in Windows reliability history.
Vegas simply stops rendering, the preview windows goes black and Vegas becomes unresponsive.
I did a memory dump from Windows Task Manager, but the filesize was over 15GB which to me, I assume is too large to send to support.

There definitely appears to be a conflict between MAGIX's AC-3 decoder when AC-3 6-channel is involved and while using Nvidia drivers later than v572.83 with a RTX5080. Yes, I know, this appears to be a very odd issue, as the audio encoding is typically handled by the CPU. Seems Nvidia have changed something in later driver releases that Vegas doesn't like in this scenario.

RogerS wrote on 8/17/2025, 9:30 AM

NVENC is only fully achievable if the NVidia GPU is selected in preferences/ video, as this GPU acceleration setting handles rendering as well as preview playback and appyling effects. But this is strangely causing the lockups during rendering with my scenario.

"Fully achievable" doesn't make sense to me. NVENC is encoding only and encodes as fast as frames are sent to it. Use no GPU or a slow GPU for the timeline? It will be waiting on frames.

Anyway 15GB may well be sendable to support- I'd open a ticket and ask. If they can use it, try Google Drive, etc.

BrettW wrote on 8/17/2025, 10:47 AM

"Fully achievable". Yes, my phrasing could have been better.

Anyway, I'll take your advice and open a ticket. Thanks