Are GPUs non-deterministic and/or how can I get them to render the exp

TTTT wrote on 6/18/2021, 2:37 AM

(Complete title was "Are GPUs non-deterministic and/or how can I get them to render the expected video" but has been truncated by forum limitation.)

Hello,

My question has rather two subquestions:

- How must of these impressions/assumptions is right or wrong?

- Is there anything I’m doing wrong, a setting I should change?

Context: Using mostly Vegas Pro, but more often noticed using plug-ins such as Ignite Pro plug-ins and recently NeatVideo.

Very often, software and plug-ins offer to use GPU in applications such as video rendering. Very often GPU rendering is much faster than CPU rendering (say for hardware of the same era and comparable "performance expectations").

What I observed is that when I use GPU rendering, OK I get videos faster but, for the most demanding renderings, or even sometimes not necessarily demanding renderings, I often get a bad rendered result (which is then also different from preview), like frame drops, turning black, image disappearing for some time, and most recently some frames getting tainted red. And I’m really talking of random results, from and rendering to another, the same frames are not necessarily wrong.

When I turn GPU rendering off so I only use CPU, I never get that kind of stuff, I get the same result pixel by pixel at each rendering .

By the way I observed this over year with various GPUs, various computers, updated or non-updated drivers, …

I’m assuming that GPU must do non-deterministic calculations. What makes me think of that is that consumer GPUs are mostly built for gaming. And the first thing that gamers watch in GPUs is: frame rate. So I’m expecting the GPUs may tolerate doing some calculation errors or approximations provided that the game flow isn’t interrupted. They might even prefer achieving "more/faster computation" than "perfectly accurate computation", as consumer will mostly judge them based on framerate.

Though, I know people and companies who use GPUs for algorithms, such as artificial intelligence, but, in my understanding, machine learning can be error resilient too. (Or it will be biased due to false negative/false positive but maybe not enough that that it’s noticeable?)

And I actually found some references explaining how programmers can actually trigger deterministic mode or not when programming with TensorFlow.

So maybe it depends on how the software/plug-ins have been written in the first place?

So how much of this is right/wrong?

How could I use GPU rendering and get accurate result for my renderings? Because I would like to benefit from faster rendering too…

Comments

Yelandkeil wrote on 6/18/2021, 3:54 AM

1,
From the last point up, you can get a faster encoding in convert programs but not in an NLE, unless you just perform cut and go render (even so, there's also circumstance to consider).
2,
If a source material is very hard coded (e.g. HEVC, XAVC whatsever) and has to be encoed again in hard codec, whereby plenty FXs with/out GPU acceleration (attention, it's acc. not encode), the thumb rule is, where the graphic card not can, it drops or jumps and go the next.
3,
you can try a "neutral" codec such as the Apple ProRes and see if there are bad frames in sight.

Generally, there's no GPU-encoding at the render output process.

It's also merely my assumptions.

TTTT wrote on 6/18/2021, 4:38 AM

Thank you for anwser.

Sorry, I just realise how encoding and rendering can be confused. (Maybe I should go over my post and check that the right word is at the right place each time. I used encoding most but has I was talking about plug-ins, I assumed it was obvious that I was talking about rendering.)

I'm not talking about encoding to compressed video formats, I can completely isolate the step in my work flow if I want to.
I'm talking of inconsistencies in video rendering, so outputting to HEVC, XAVC or uncompressed AVI doesn't make a difference.

Yelandkeil wrote on 6/18/2021, 5:03 AM

Now it's clear! And that's why I mentioned the ProRes codec.

Let's take the HEVC as an example.

If you have proper hardware you should have two alternatives of them, the one is remarked with its hardware name (or something like that), if you choose this one, it really encodes the signal from timeline into HEVC. But if it's a weak graphic card it could be slower that you use CPU only.

If you take the "pure" one, then CPU encodes the signal into HEVC. But your GPU helps by the computing for FXs and so on (so called hardware acceleration).

Now I'm also confused into the enconding or rendering....😇

RogerS wrote on 6/18/2021, 6:29 AM

It's something with how Vegas reads ahead preview data when doing GPU assisted encoding in my experience. I have never seen glitches except when using GPU accelerated encoding templates like NVENC.

Turn dynamic ram preview off (set to 0MB) and these glitches should disappear. If they don't you may need to disable GPU preview acceleration as well. Not the GPU's fault but unhappy combinations of old and new features.

Happy to test a simple project if you have one available.

TTTT wrote on 6/18/2021, 7:23 AM

I currently get the problem when rendering uncompressed .avi (4K 29.97 fps).

(If I remember well NVENC has disappeared for me and encoding settings that I used to use are completely buggy since version 2018 and the only encoding setting that I can used to export compressed video now is Magix HEVC encoded with Intel QSV. But again, encoding has it's share of problems but again, it is not related to the issues I'm talking about here.)

I have tried many parameters including turning dynamic ram to 0 or to high values, not change noticed.

I know that I can turn GPU of (and I also need to turn GPU off in NeatVideo settings has the plugin has it's own settings) but this I'm working on a long projet that is painful to render in preview window so I do short renders to files very often. And this is just multiplying the time I have to wait to get each sample...

It's also kind of frustrating because my current computer, from 2019, was the most expensive I ever assembled/bought, but regarding creative software use I feel crippled comparing to video editing 10 or 15 years ago ... And solutions offered are very often like "oh turn that off", "oh don't use that", "oh that's because the new version is just slower"...

JN- wrote on 6/18/2021, 4:57 PM

@TTTT Can you give us concrete information on your VP version and build, PC machines specs, the mediainfo of the video you are encoding, a sample clip(s) or small project that causes this problem. Other users can then test same to reproduce similar errors or not. Also supply screenshots of project properties and the specific render template in use.

Otherwise this becomes just a general discussion about encoding/rendering, but is unlikely to help in fixing your issues.

This is an issue you report … “turning black, image disappearing for some time, and most recently some frames getting tainted red.” I have had that with specific media types recently and Magix ….. (can’t remember the full name) gave a solution. No doubt what assisted in the developer contributing his solution was the very specific information that I supplied about the VP version number, build and media type. My PC specs were already in my signature.

OTOH if your intent is a general discussion then by all means carry on.

studio-4 wrote on 6/19/2021, 1:57 AM

When researching various GPUs for upgrading the cheap video card in my Xeon PC, I noticed a number of references to NVIDIA's TU116 "Turing" GPU (but not the 117 for some reason), and it being particularly good for NLEs. I assume the Turing part is for stochastic-analysis/prediction of common video processes, but still unclear on the specifics. The only relevant article I found discussed the machine-learning products NVIDIA is marketing for large-scale server implementations.

RogerS wrote on 6/19/2021, 2:58 AM

That's the 1650 and NVIDIA of a similar release date. Turning is just the generation of hardware architecture. It's not doing anything with machine learning in Vegas. This generation of NVIDIA is cheap and works well enough in Vegas (and may actually be available?)
https://www.techpowerup.com/gpu-specs/nvidia-tu117.g881
https://www.techpowerup.com/gpu-specs/nvidia-tu116.g902
The 117 appears to be the same generation but significantly scaled down.

the only encoding setting that I can used to export compressed video now is Magix HEVC encoded with Intel QSV.

If you want actual advice please provide the info (and sample files) JN- requested so we can test using a variety of hardware.

That suggests something quite wrong with your system configuration. I'd do a reset of Vegas (control-shift on startup) and a full uninstall/reinstall of GPU drivers before reinstalling Vegas.

What GPU and CPU are you using exactly?

studio-4 wrote on 6/19/2021, 3:23 AM

That's the 1650 and NVIDIA of a similar release date. Turning is just the generation of hardware architecture. It's not doing anything with machine learning in Vegas. This generation of NVIDIA is cheap and works well enough in Vegas (and may actually be available?)
https://www.techpowerup.com/gpu-specs/nvidia-tu117.g881
https://www.techpowerup.com/gpu-specs/nvidia-tu116.g902
The 117 appears to be the same generation but significantly scaled down . . .

@RogerS

Thanks for clearing that up! It was more idle curiosity than anything else. I assume your second paragraph refers to the OP's question, but thanks for answering mine! (And thanks for the links!)

For the record, I have a cheap AMD Radeon R7200 (memory unknown) that the vendor threw in for free. Windows' device manager only tells me the model number, but not the amount of installed VRAM.

Unfortunately, I can't use Vegas' slow-motion effect at all with it (it says I don't have enough VRAM to complete the action). That's why I was in a hurry to buy a new GPU during literally the worst time in human history to try to buy a GPU.

So after several days of searching online, I found an AMD Radeon RX470 4GB GPU on eBay which I sniped for $200. It was the only card I could find that wasn't 4x it original MAP, and pretty much the only GPU I could find for around $200. It was supposed to be delivered today, but it wasn't, and I'm super disappointed, 'cause you know, it's Friday and I was excited to play with it over the weekend. Hopefully, tomorrow. Anyhow, thanks!

RogerS wrote on 6/19/2021, 3:27 AM

Yes, second paragraph was for the original poster. I edited and tried to make that clearer.

Here are the current system requirements. You don't want to upgrade Vegas after >5 years after your hardware's release as support will be dropped for older hardware.

https://www.vegascreativesoftware.com/us/vegas-pro/specifications/#productMenu

System requirements

Operating system: Microsoft® Windows 10 (64-bit)

Processor: 6th Generation Intel Core i5 (or AMD equivalent) or better. 2.5 Ghz and 4 Core minimum. For 4K, 7th Generation Intel Core i7 (or AMD equivalent) or better. 3.0 Ghz and 8 Core minimum.

RAM: 8 GB RAM minimum (16 GB recommended; 32 GB recommended for 4K)

Hard drive space: 1.5 GB hard-disk space for program installation; Solid-state disk (SSD) or high-speed multi-disk RAID for 4K media

Internet connection: Required for downloading, registering and validating the program, as well as for some program functions.

Program languages

English, Deutsch, Español, Français

Supported GPU

NVIDIA

NVIDIA® GeForce GTX or RTX 9XX series or higher with 4GB

AMD/ATI

AMD/ATI® Radeon with 4GB and VCE 3.0 or higher (Radeon Pro series with 8GB for HDR and 32 bit projects)

Intel

Intel® GPU HD Graphics 530 series or higher

studio-4 wrote on 6/19/2021, 3:46 AM

NVIDIA® GeForce GTX or RTX 9XX series or higher with 4GB

Thanks! Damn! I did read the Magix minimum requirements, but I think I saw that after I ordered the Radeon RX470 4GB GPU, so I'm unsure exactly where that ranks. It has to be notably faster than the cheap adapter I have in there now, judging by size alone (it's a pretty small card). I chose the RX470 partly because I was able to find another member here that has that same card and uses it without issue on Vegas Pro 16.

The other components in my circa-2015 retired server (purchased used a couple weeks ago) appear to meet or exceed the other minimum specs. My full system specs are in my signature, but here's the short version:

• HP Z440 Intel Xeon E5-1650 v3 3.5GHz quad-core CPU.
• 32GB ECC RAM.
• 1TB SATA SSD.

RogerS wrote on 6/19/2021, 3:51 AM

@studio-4 Well, you needed to scroll down to AMD right below it. For ordinary 8-bit projects its fine for now, but as a 2016 card may not be supported in another year or two. For a user sticking with VP 16 that's not an issue.

Specs:

This site says it is VCE 3.4: https://www.techpowerup.com/gpu-specs/radeon-rx-470.c2861

studio-4 wrote on 6/19/2021, 4:08 AM

Oh, cool! Thanks! I think I actually checked that site, pre-purchase. Yeah, I got the AMD and NVIDIA brands mixed up. I'm only editing 8-bit 1080p24 for the foreseeable future. All of my cameras (in which I have a fair amount invested) are only 1080-capable, so I won't be shooting 4K anytime soon.

Hopefully the RX470 will tide me over until the GPU craziness subsides a bit. If not, it looks like I can turn around and sell it for what I paid (at least for the time being). The only spec I made sure the RX460 supported is OpenCL (which it does, v2.1).

Musicvid wrote on 6/19/2021, 5:44 AM

What I observed is that when I use GPU rendering, OK I get videos faster but, for the most demanding renderings, or even sometimes not necessarily demanding renderings, I often get a bad rendered result

That is absolutely correct.

Rule number one. Size, Quality, Speed -- pick only two!

That is the only "deterministic" outcome of machine rendering, except you only get to pick speed.

All the AI in the universe will never change rule #1. You cannot artificially replace lost quality, you can only make it less, even if we are easily fooled by appearances in some cases.

TTTT wrote on 6/19/2021, 6:24 AM

@TTTT Can you give us concrete information on your VP version and build, ...

Well this is more like a general question about GPU and rendering with various plug-ins. I feel like the issue in on GPU processing side rather than in Vegas, considering that I faced various issues with GPU encoding every now and then for years. (But always using Vegas Pro and no tried on other softwares.)

Current build I use is 18.0 (build 527), GPU is GTX 1080 TI, RAM is 32 GB. Currently seems the random-frame-issues related to using NeatVideo for Vegas. I also had to remove one of the Ignite Pro plug-ins because that one cause sequences disappearing, but no randomly. Current source file is an uncompressed .avi 1920x1080 29.97i placed on Samsung 980 Pro which is my fastest drive (as it's a ~175 GB file). Output 2880x2160 29.97p.
But as I said I already had similar issues with pas configurations, if I remebmer well, Intel I7 6700K, GTX 980, RAM 32 GB too, must have been Vegas Pro 16 or 17 and Ignite plug-ins ...

JN- wrote on 6/19/2021, 6:39 AM

@TTTT "Well this is more like a general question about GPU and rendering with various plug-ins. " Ok, fair enough. Nothing wrong at all in talking about these things, I enjoy that. Not to be confused with identifying and fixing an issue.

If you change your mind you can supply a sample project or clip that does cause you problems. A good few users here use NV, including myself, so you can include that in the project.

TTTT wrote on 6/19/2021, 6:47 AM

the only encoding setting that I can used to export compressed video now is Magix HEVC encoded with Intel QSV.

If you want actual advice please provide the info (and sample files) JN- requested so we can test using a variety of hardware.

That suggests something quite wrong with your system configuration. I'd do a reset of Vegas (control-shift on startup) and a full uninstall/reinstall of GPU drivers before reinstalling Vegas.

I'm quite sure that I already tried that, but I may try it again (when I get time for this again).
But regarding the encoding options I guess I rather ask is in another thread as I see don't see it as related to this topic.
It's just that we started confusing rendering and encoding to compressed format, but as you can see in my posts here, I'm inputting uncompressed .avi and outputting uncompressed .avi. (And chancing the input or output doesn't seem to change anything.)

RogerS wrote on 6/19/2021, 6:58 AM

I'm quite sure that I already tried that, but I may try it again (when I get time for this again).
But regarding the encoding options I guess I rather ask is in another thread as I see don't see it as related to this topic.
It's just that we started confusing rendering and encoding to compressed format, but as you can see in my posts here, I'm inputting uncompressed .avi and outputting uncompressed .avi. (And chancing the input or output doesn't seem to change anything.)

So far this situation is the only example I've seen of such issues going from uncompressed AVI to AVI in a few years on this forum and two Vegas Facebook groups.

I'd like to test it and see if anyone can replicate it on a system other than yours. My guess is that there is something very wrong with this configuration (likely GPU) and Vegas, but guess are just that without verification through controlled experiments.

TTTT wrote on 6/19/2021, 7:16 AM

I think I havn't been very precise on this but when I say I need to turn off GPU, I mean that I turn of GPU in NeatVideo setting (it's a plug-in but it's got its own settings). Vegas GPU is currently active without issue.

RogerS wrote on 6/19/2021, 8:39 AM

Okay, what are you trying to test? Neat Video itself isn't working properly? I have not seen that.

I also use Neat Video with GPU enabled, and have VP 15-18 here with an integrated Intel and separate NVIDIA GPU. If you want to see if yours is not the only system affected please construct a sample test.

Howard-Vigorita wrote on 6/19/2021, 12:26 PM

Neat video has issues with certain amd boards and newer drivers. As their drivers are unified, I wouldn't be suprised if they affect more that the few boards listed. https://www.neatvideo.com/support/known-issues/gpu

Btw, the rx480 is probably the bottom of the amd line for Vegas functionality. Suggest you keep an eye out for a deal on ebay for an rx580 or 590. When I upgraded from a 480 to a 580 a number of years ago, the performance increase in Vegas was substantial.

TTTT wrote on 6/20/2021, 11:37 PM

Rule number one. Size, Quality, Speed -- pick only two!

That is the only "deterministic" outcome of machine rendering, except you only get to pick speed.

Re-thinking about this... I think there's some shortcut that is taken in this assumption (though it cab be up to interpretation).
A CPU will always give the same result for a given computation. Let's say simplify and replace rendering with adding numbers, pretending that it's a demanding task.
Result of 2 + 2 on a CPU will always be 4. No matter the load of the CPU.
My impression and my initial question is that I fell like for GPUs, especially if they are overloaded or so, it's acceptable to reply once that 2+2 is 4.0001 and on another occasion that 2+2 is 3.999998 ... or possibly to skip answer to 2+2 and go on with the next computation. That's what I called non-deterministic.

Coming back to video rendering, with CPU with the same input and parameters, I always get the same result, bit to bit. With GPU ... it looks like ... "it depends".

RogerS wrote on 6/21/2021, 1:11 AM

GPU can take shortcuts for gaming and the like, but the integer calculations should be always identical. Floating point math may not be so and NVIDIA talks about double calculations for high precision applications.

Tensor Cores accelerate HPC

The performance needs of HPC applications are growing rapidly. Many applications from a wide range of scientific and research disciplines rely on double precision (FP64) computations.

To meet the rapidly growing compute needs of HPC computing, the A100 GPU supports Tensor operations that accelerate IEEE-compliant FP64 computations, delivering up to 2.5x the FP64 performance of the NVIDIA Tesla V100 GPU.

The new double precision matrix multiply-add instruction on A100 replaces eight DFMA instructions on V100, reducing instruction fetches, scheduling overhead, register reads, datapath power, and shared memory read bandwidth.

Each SM in A100 computes a total of 64 FP64 FMA operations/clock (or 128 FP64 operations/clock), which is twice the throughput of Tesla V100. The A100 Tensor Core GPU with 108 SMs delivers a peak FP64 throughput of 19.5 TFLOPS, which is 2.5x that of Tesla V100.

With support for these new formats, the A100 Tensor Cores can be used to accelerate HPC workloads, iterative solvers, and various new AI algorithms.

https://developer.nvidia.com/blog/nvidia-ampere-architecture-in-depth/

For your issue it's likely just a driver problem/software bug.

TTTT wrote on 6/21/2021, 1:17 AM

I was using addition as an example ... I have not idea what video treatment algorithms use, I have not idea if they use integer, floating point or else, but I assume it depends how they were written.