New machine, always gets render error 0x8066000a when using NVENC

RealityStudio wrote on 3/22/2021, 7:26 PM

Hey everyone I got a new laptop, an Asus Rog Flow X13 to use with Vegas Pro 18 and unfortunately have run into a snag that I just can't figure out. Basic specs are an 8 core Ryzen Cpu, NVidia 1650 gpu, 16GB ram, Vegas Pro 18, latest of everything software and driver wise, using NVidia studio driver. Source video is 4k.

Main issue is it will give error 0x8066000a anytime I try and render a video, I tried using the 4K NVenc presets. Rendering with the 720p and 1080p NVenc presets actually does work, it's just the 4K preset that gives the error which of course is the one I need.

My desktop pc which has an NVidia 1080ti can encode with the 4k NVenc preset just fine on the identical footage. I thought the 1650 was Vegas compatible but now I'm thinking maybe it's not? Most benchmarks I see on this machine are with Premiere Pro and Davinci Resolve, never seen anyone try Vegas Pro on it.

Anyways I'm at a loss, any help appreciated!

 

Comments

RogerS wrote on 3/22/2021, 7:35 PM

Sure the 1650 is generally speaking compatible with Vegas. 1660ti has been benchmarked here

Which preset is it? MagixAVC? Try the high quality one rather than lossless? Is dynamic ram preview set to 0?

RealityStudio wrote on 3/22/2021, 7:38 PM

I tried both 0 and the default 200MB for dynamic ram preview, same result. I went with the MagixAVC preset Internet UHD 2160p 29.97fps (NVENC). The 1080p and 720p versions of that preset work fine, the 4k one gives the error. Creepy part is this error code seems to go back some years from a google search, and I've never seen a resolution for it :(

JN- wrote on 3/22/2021, 8:36 PM

@RealityStudio Try this. Don’t use a premade, already in place render template preset. If that doesn’t work, do use an already in place render template.

 

Whether its a premade render template or one you make up yourself ...

#1 Don't use the default and VBR, ... below ...

 

#2 Do use HQ and vbr - HQ ... below ...

 

If that doesn’t work, you need to give us a screenshot of the render template you are using. Also, I take it it is just Nvenc avc, not hevc nvenc you are attempting.

Last changed by JN- on 3/23/2021, 4:27 AM, changed a total of 2 times.

---------------------------------------------

VFR2CFR, Variable frame rate to Constant frame rate link to zip here.

Copies Video Converts Audio to AAC, link to zip here.

Convert 2 Lossless, link to ZIP here.

Convert Odd 2 Even (frame size), link to ZIP here

Benchmarking Continued thread + link to zip here

Codec Render Quality tables zip

---------------------------------------------

PC ... Corsair case, own build ...

CPU .. i9 9900K, iGpu UHD 630

Memory .. 32GB DDR4

Graphics card .. MSI RTX 2080 ti

Graphics driver .. latest studio

PSU .. Corsair 850i

Mboard .. Asus Z390 Code

 

Laptop… XMG

i9-11900k, iGpu n/a

Memory 64GB DDR4

Graphics card … Laptop RTX 3080

RogerS wrote on 3/22/2021, 8:43 PM

FWIW, it just worked here with a 4K 24p source to 4K 24p file using NVENC on my GTX 1050 card using these settings:

RealityStudio wrote on 3/22/2021, 8:49 PM

So I did try both premade and my own customized presets, including toying with VBR/CBR, trying really low bitrates, quality presets, no luck yet. Yeah this is for h.264, not HEVC. I attached my custom 4k preset from my desktop pc, which I also tried recreating identically on the laptop. Works fine on the desktop but not the laptop, render preset specs in attached pic.

RealityStudio wrote on 3/23/2021, 1:02 AM

Ok I'm starting to suspect there is something about this laptop that just isn't compatible with Vegas Pro, perhaps some memory limit. If I take the default Magix h.264 NVEnc 4k preset and change *only* the resolution, I can get it to render clips up to 3840x1440. At that resolution it renders perfectly fine with the stock preset and just the resolution changed. Bump that up to 3840x1600 though and it "renders" but video is all black, although you can still hear audio. Then bumping that to actual 4k at 3840x2160 causes the error to appear and no render occurs.

My best guess is that it's some kind of memory limitation, and while Premiere Pro and Davinci Resolve are ok with it as I've seen many Youtube videos benchmark both of those apps on this exact spec laptop, Vegas Pro breaks with top resolution to render being 3840x1440.

I'm going to give up and just assume Vegas Pro is not compatible with the laptop, I'll just return the laptop to Asus and move on. Thanks for the help.

RogerS wrote on 3/23/2021, 1:37 AM

Maybe file a support request to Vegas Creative Software on this issue. I've not seen this before- very interesting it's only a problem with the resolution.

fr0sty wrote on 3/23/2021, 2:16 AM

If you want a good asus laptop for VEGAS, the zenbook pro duo has been an absolute dream machine for me. I love this thing. 2 4k touch screens and tons of power.

RealityStudio wrote on 3/23/2021, 2:39 AM

I actually had bought two laptops, that Asus Rog Flow X13 and the Hp Spectre x360 14". Basically I wanted 3:2 aspect ratio and a two in one so I can flip the screen and use it tablet style. The Asus unfortunately didn't work out, I've already factory reset it and will rma it tomorrow. The HP on the other hand I've been testing tonight and it's been working great with Vegas, I get 30fps on the timeline with my 4k video + Lut + Watermark so looks like I'll keep that one. And good lord is that oled display nice :)

One interesting thing I noticed, normally when encoding in Vegas Pro with NVEnc you'll see it render, then stall, then render, etc the entire way through. The HP laptop I got is Intel Quicksync and it doesn't do that. When I render with a Magix QSV preset it just renders non stop with no stalls. Quite interesting.

RogerS wrote on 3/23/2021, 3:24 AM

Does it render better or worse through Vegas with Voukoder? There Vegas just sends Voukoder frames and it handles the encoding with NVENC or the CPU (high quality x.264).

JN- wrote on 3/23/2021, 4:31 AM

@RealityStudio Have you used the RC mode and Preset I suggested in #2 ?

The reason I ask is because the screenshot that you posted is the #1 settings I suggested that you don't use.

Last changed by JN- on 3/23/2021, 4:36 AM, changed a total of 1 times.

---------------------------------------------

VFR2CFR, Variable frame rate to Constant frame rate link to zip here.

Copies Video Converts Audio to AAC, link to zip here.

Convert 2 Lossless, link to ZIP here.

Convert Odd 2 Even (frame size), link to ZIP here

Benchmarking Continued thread + link to zip here

Codec Render Quality tables zip

---------------------------------------------

PC ... Corsair case, own build ...

CPU .. i9 9900K, iGpu UHD 630

Memory .. 32GB DDR4

Graphics card .. MSI RTX 2080 ti

Graphics driver .. latest studio

PSU .. Corsair 850i

Mboard .. Asus Z390 Code

 

Laptop… XMG

i9-11900k, iGpu n/a

Memory 64GB DDR4

Graphics card … Laptop RTX 3080

RealityStudio wrote on 3/23/2021, 1:37 PM

Yeah I did mess with all the NV Encoder, VBR, etc settings. I'm wonder if it's because that particular Asus laptop is a curious beast. It has an AMD Radeon gpu on the cpu die, it also has an NVidia 1650 dedicated cpu, but thirdly it has an external custom 8 lane pcie connector so you can attach their small external 3070 or 3080 gpu (which I didn't have). I wonder if because of this they had to do some funky business in the hardware setup internally to get it all to work? No idea.

It was still worth doing all this with both laptops as I learned a nice render speedup in the process. When I was testing the HP laptop it could not display my 4k + Filmconvert footage at 30fps on the timeline, the Intel Xe gpu could muster 23 fps at most. So I stepped away from that one and moved to testing the Asus. In the process of debugging that one I learned that Vegas Pro can save out a LUT really easily. Long story short, apply all my Filmconvert settings then have Vegas Pro save that as a LUT. Remove Filmconvert from the FX chain and going forwards I only need to apply my LUT to get the identical look but with *much* faster performance. Result is now the HP laptop can play my 4k + LUT footage no problem at full 30fps. Side bonus, renders are now 35% faster as well. Double side bonus, now I can keep the laptop with the oled screen.

I only tried Voukoder once, it wasn't suitable for my needs but the Intel QSV render was rendering like Voukoder did in the sense that there were no stalls.