FREQUENT rendering errors with NVENC Encoder

SamJoex wrote on 1/9/2021, 3:41 AM

I am having immense trouble getting the NVENC Encoder to render my videos properly (or at all) on my new laptop. Many times, it produces inexplicable glitched frames and/or fails to render altogether (freezing the render progress forever). Pressing Cancel softlocks VEGAS Pro but doesn't crash it, forcing me to Task Manager End Process it. On projects with barely any resource-intensive items (such as high-res images and videos), it sometimes renders ok, but in other projects it routinely fails.

Discrete GPU: NVIDIA GTX 1650 (Notebook)
Video Driver: NVIDIA Studio Driver 460.89
Video Editor: VEGAS Pro 15.0 Suite (Build 416)
(other computer specs in signature)

I am rendering my videos in 4K 60FPS, but it tends to fail at lower resolutions as well.

I've found Mainconcept AVC to be a way more reliable encoder, but it doesn't use the NVIDIA GPU (and instead uses my Intel i5-10300H CPU exclusively), operates much slower, and generally produces larger file sizes. And while the better Preview experience is nice, I'd like to use the discrete GPU for making renders faster and more efficient as well.

I suspect the problem lies somewhere within VEGAS Pro 15 or the NVIDIA driver.

Comments

Grazie wrote on 1/9/2021, 4:51 AM

@SamJoex - Could be that you may have a Clash of nVidia card acceleration. Do you have any nVidia accel compliant FXs/Events needing nVidia? As you have found other Render Template works fine this maybe the issue here.

j-v wrote on 1/9/2021, 4:53 AM

@SamJoex

Please show here a screenshot of your settings at Options/Preferences/Video and which type of driver is installed for your Nvidia?
Mind that the drivernumber of the latest is the same for Gamedriver and Studio driver but both are different.

RogerS wrote on 1/9/2021, 6:10 AM

Set dynamic ram preview to 0Mb and you may have to turn off GPU acceleration in preferences. I had the same issues with 15.

SamJoex wrote on 1/9/2021, 3:08 PM

@j-v

j-v wrote on 1/9/2021, 3:18 PM

For rendering your amount of reserved memory is much to high, set it to the default 200 or to 0

SamJoex wrote on 1/9/2021, 6:19 PM

OK, after disabling GPU Acceleration and setting Dynamic RAM Preview to 0 MB, the render went much better, and succeeded. However, there were at least 3 places in the video file that contained glitched frames (insomuch that they must be re-rendered), and the rendering process slowed down quite significantly in many areas (although I'd rather have that than a failed or glitched render).

SamJoex wrote on 1/9/2021, 6:39 PM

UPDATE: I re-rendered one of the sections of the video that glitched, using two different NVENC Encoder presets. Under the "Default" preset, the render gltiched like before. However, under the "High performance" preset, it not only rendered without glitches, but it also shrunk the file size of the video portion from the high 400s MB to 366 MB.

Now to see if this works for the entire video.

(Also I set the Dynamic RAM Preview up to 200 MB, as I heard from another forum post that it was the sweet spot.)

JN- wrote on 1/9/2021, 8:06 PM

@SamJoex I have found that not all nvenc encoder settings are created equal!. The following table shows some of the combinations that I tested, for quality etc. Note that while item 11 returned a good quality, it will produce very slow seek times. I personally only ever use items 2 or 7.

SamJoex wrote on 1/9/2021, 10:17 PM

I rendered the video with NVENC Encoder with Preset being High Performance and RC Mode being CBR, and it rendered without flaw.

Thanks for your help, everyone!