Benchmark Roundup 2021

Howard-Vigorita wrote on 6/28/2021, 3:21 AM

Well I finally got around to updating my Benchmarking methodology and am rolling out a new online Roundup21.

I've automated the benchmarking itself with a script that's still a work in progress. I've never done an application with scripts before but I'm learning. So far I've knocked out running the Sample Project benchmark as well as the original RedCar and the 4K AVC, HEVC and ProRes422 extensions. I've also updated the online Magic Table html to hopefully make it more useful as it grows in size.

So far I've tested my scripting system on 1 machine and here are the results:

http://www.rtpress.com/Roundup21.htm

For a graphical presentation, I also did a little chart:

This is my quickest machine. The build is documented here. As I knock out benches on my other machines, the contrast should be interesting. What I found interesting on this machine is that the Sample Project benchmark shows minimal variation between the accelerated VCE and QSV encoders compared to the CPU-based MainConcept encoder. While the RedCar variations show significant contrast for HD and dramatic contrast for 4K camera footage. Keep in mind that this machine has pretty high performance CPU and GPU. More to follow.

Comments

RogerS wrote on 6/28/2021, 3:55 AM

Really interesting, thanks. QSV holds its own.

Is this something where others could run it with the same script/program and generate results in the future?

Howard-Vigorita wrote on 6/28/2021, 12:18 PM

Absolutely. The zips are linked to a google drive in my signature. The ProRes one doesn't have exactly the same media in it, however. I think I put a shrunk-down ProResLT in instead... only around 5 gb. The 21 gb ProRes422 clip wouldn't fit in my free 15 gig drive. But it's easy enough to make one with vp18 using the Sony mxf clip from the original... there's a doc page there that mentions that. I'm contemplating a way to do one with ProResRaw too.. if and when vp19 releases with it. Fingers crossed on that.

Howard-Vigorita wrote on 6/29/2021, 11:32 PM

Successfully ran the scripting system overnight on my Xeon system. That's my b-system which is a little older and slower than my 9900k one. It doesn't have an onboard igpu but the 5700xt little Navi gpu seems to have improved with driver and Vegas updates. To help make up for the lack of an igpu, I added in an Nvidia 1660 devoted to decoding. I'll be rerunning benches with Vegas 16 and 17 at some point to nail down if the improvements are all from Vegas 18 or in AMD drivers. Here's a screen shot of the charts comparing the systems stacked for easy comparison:

The most interesting thing to me is that the Sample Project on the slower Xeon system benches better than my fastest one. I think that's solely attributable to 1 thing: the Sample Project benchmark gets a large proportion of its load from a Vegas fractal noise media generator and the 5700xt happens to do that one thing extraordinarily well.

The other thing I find noteworthy is how closely bunched the benches which use Avc, Hevc, and ProRes media are. Considering that Avc only benefits minimally from hardware decoding, Hevc depends on it, and ProRes isn't supported directly by hardware decoders. I have a suspicion that the Apple library licensed by Vegas is using the gpu for number crunching, much like mining apps do... 3D video utilization looks unusually high when it's running.

Still working on the script but its shaping up. Hoping to eventually share it, perhaps for tutorial purposes. Or do a video on the finer points of its development. It's derived, by the way, from the excellent work of @jetdv whose tutorial videos were invaluable... I should mention that I've never authored a Vegas scripting app till this past week or even used Visual Studio before.

RogerS wrote on 6/29/2021, 11:40 PM

Very interesting, keep us apprised as the script develops. It's hard to know what Fx to apply that challenge the hardware but are also fair overall for comparison. TechGage was using MedianFx. Likely a diversity of projects is the most useful and will help tease out which hardware excels at what.

Howard-Vigorita wrote on 6/30/2021, 8:37 AM

That's an interesting idea... benching some individual FX. But performance with one FX is not necessarily indicative of performance in others so the key is probably including a variety. Preferably ones commonly used by most users. I recall one of the reviewers focusing once on LUT performance and found some gpus did it better than others. That's becoming a pretty important one.

jetdv wrote on 6/30/2021, 8:48 AM

@Howard-Vigorita, glad to hear my tutorials are helping you!

Howard-Vigorita wrote on 6/30/2021, 9:17 AM

👍 I'll throw a little about it into the Scripts section...

FernC wrote on 6/30/2021, 9:26 AM

Thanks for all the ideas, I"ll take a look it tomorrow and try some things out, and read the comments more thoroughly . when I first tried that project I was using Nvidia drivers 1 release before current, then updated to current release, both studio versions. After not having success with rendering and afterwards crashing, I removed the ogg audiio file, on project load instead of filling 8GB of Video card memory it almost filled which I still don't think is normal?. It then allowed me to render and not crash. After that success I re-linked the audio file in it's new location, and the render worked fine. I then replaced audio and loaded complete project and this time it loaded, rendered, and did not crash which I wasn't expecting.

This not working, then working makes me a bit suspicious it may be my system, and something is using a small about of GPU memory which is causing the faults, and other times I"M lucky and the GPU memory is completely empty before Vegas Project load. I must look into that

Howard-Vigorita wrote on 6/30/2021, 10:16 AM

Funny that the ogg file would have that impact... I wouldn't have expected that. I'm more inclined to expect that the Noise Generator might be the culprit... which would be good news if the only gotcha was having to avoid that fx in your work.

Thanks for the feedback, btw. I'm interested in eventually getting a 3000-series card myself and use the benchmarks to help guide my purchasing decisions.

RedRob-CandlelightProdctns wrote on 7/1/2021, 12:38 PM

Most of my renders I still render to FHS (1920x1080) MP4s using MainConcept encoding. Looking at your results I'm now wondering if I ought to use a different CODEC to generate that same format, noting how much slower the MainConcept one is?

Vegas 21.300

My PC (for finishing):

Cyperpower PC Intel Core i7-7700K CPU @ 4.2GHz, 64GB mem @ 2133MHz RAM, AMD Radeon RX470 (4GB dedicated) with driver recommended by Vegas Updater (reports as 30.0.15021.11005 dated 4/28/22), and Intel HD Graphics 630 driver version 31.0.101.2112 dated 7/21/22 w/16GB shared memory. Windows 10 Pro 64bit version 10.0.19045 Build 19045.

My main editing laptop:

Dell G15 Special Edition 5521, Bios 1.12 9/13/22, Windows 11 22H2 (10.0.22621)

12th Gen Intel Core i7-12700H (14 cores, 20 logical processors), 32 GB DDR5 4800MHz RAM, Intel Iris Xe Graphics, NVIDIA GeForce RTX 3070 Ti Laptop GPU w/8GB GDDR6 RAM, Realtek Audio

 

 

Howard-Vigorita wrote on 7/1/2021, 1:34 PM

I think you need to balance that with a quality assessment. Most folks here seem to subjectively like the look of the MainConcept encode better. Also, I did get complaints once upon a time from an Nvidia user who was not able to run with a VCE rendered mp4. But both of those things may just be past history cured by Vegas and driver updates all around. Recent quality analysis I've been doing seems to show many formats with better objective VMAF quality than MainConcept, particularly Voukoder. But I haven't done as many MC's yet because they take so long...I probably need to take a bit more time with them to see if there's some combo of settings with a superior sweet spot . Here's a link to the quality tables I've run so far:

http://www.rtpress.com/ffmetrics.htm

Howard-Vigorita wrote on 7/1/2021, 2:29 PM

Got more overnight Render benchmark updates... one for my Dell xps15-9570 laptop and the other for my Intel Hades Canyon NUC. Both of these have the same sized (32 gigs) RAM and the same Intel igpu. Alhough I used some pretty fast RAM building the NUC. The NUC cpu only has 4 cores (compared to 6 for the xps15) but shows the same processor boost speeds (3.9 ghz per CPUz) as the xps15 while running the benches suggesting that their fan/cooling profiles are very similar. Here's how they stack up to each other:

Rendering to FHD 1080p frames, the NUC shows small advantages across the board. Doing 4K 2060p renders, the MainConcept encoder, which employs CPU-only encoding, is noticeably faster on the NUC... probably because being relieved of rendering duties, the GPU performance for everything else goes up and becomes more significant; which would include decoding, fx, track mixing, and other timeline processing. All of which the NUC Vega M is faster at than the xps15 Nvidia 1050ti. This also gets reflected in Vegas editing performance where the NUC performs much more briskly which is the main reason I go to the trouble of jamming it into my carry-on for work on the road.

IAM4UK wrote on 7/2/2021, 6:52 PM

Interesting; I will try to benchmark my i7-11700K with VEGA64 Frontier.
Meanwhile, my first test with a simple render of UHD, HEVC, QSV versus VCE (both at high/best quality) showed that VCE was SIX TIMES FASTER than QSV on my system. That surprised me, but my previous CPU didn't have any integrated graphics, so I am new to QSV...

IAM4UK wrote on 7/2/2021, 7:52 PM

I obviously don't know the approved way to run the benchmark renders. I got ONE of the results that makes sense, and is comparable to your i9-9900K (which my i7-11700K with VEGA64 Frontier should be). The rest of my results made no sense...
Red Car 4K rendered to AVC UHD with VCE: 53 seconds (comparable to your fastest result)

Red Car 4K rendered to AVC UHD with CPU: 3 minutes, 3 seconds

(QSV is not an option for AVC renders on my machine in VP18.0.0.527))

Red Car HEVC rendered to HEVC UHD with QSV: 6 minutes, 38 seconds

Red Car HEVC rendered to HEVC UHD with VCE: 4 minutes, 22 seconds

(CPU is not an option for HEVC renders on my machine in VP18.0.0.527)

Howard-Vigorita wrote on 7/3/2021, 10:06 AM

The 11700K has the newer Intel uhd750 igpu in it which is one of the Xe/Iris-line igpus that's supposed to be faster than any of the hd630 ones that I have. Vegas 18 is supposed to support it. You might want to verify that you don't have an 11700F or 11700KF... those omit the igpu.

I'd suggest you first make sure your Intel graphics driver is up to date by doing a driver check on the Help drop-down. It should show there with .9466 as the latest version as of this moment. If it's missing there, its probably not on your preferences-video tab either.

See if it shows up as a gpu on the preferences-video tab. The Amd should be selected there as your primary GPU and be flagged as Optimal. If the Intel is absent as a #2 selection choice, that means it's not detected by Vegas and you may have to go into your motherboard bios to enable it. I would expect your motherboard to enable it by default, however.

If it's detected properly on the graphics tab, check the preferences-IO tab and see that the Intel igpu is selected for decoding. I expect it would be flagged as Auto there but you can select it manually if it's not. Also make sure none of the legacy boxes are checked there.

You might also double-check the General tab and make sure "Use Intel QSV for encoding and decoding if available" is checked. Usually Vegas checks that box by default.

Another thing you can check with is the free GPU-Z app from Tech Power-Up. It's a handy utility to detect your gpus independent of Vegas. It lists gpu's, display driver version info, and also displays which driver api's are implemented. The Intel driver should have the OPENCL api box checked. If it doesn't, you may need to do a clean uninstall of Intel drivers (get the free DDU app from guru3d) so you can install the older Intel driver supplied by your motherboard maker first, before updating with a newer generic Intel driver. This used to be a big issue with Intel drivers but not lately.

Howard-Vigorita wrote on 7/3/2021, 11:14 AM

Btw, all of my RedCar renders are done with the default Magix AVC preset parameters (1080p or 2060p) at the same 29.97 frame rate as the originally supplied media. They look like this on my laptop:

This contrasts with the SampleProject whose media is supplied 25 fps and should be rendered at that rate. See the presets as specified in the Benchmarking Continued thread... noting that the SampleProject UHD preset specifies non-default parameters which need to be adjusted manually... I save each of these as a custom preset for convenience and to avoid making mistakes.

Howard-Vigorita wrote on 7/4/2021, 9:56 PM

Finally got my over 10-year old 980x system up to speed with win10 updates. It was originally win7 till last year when also I swapped out its old rx480 for a Vega64 gpu. This machine has no igpu. But with the nice gpu, it clearly runs Vegas better than my laptop. Here's how the benches compare with my xps15 and NUC machines.

In the FHD Sample Project it outperforms both of those machines. And as with most of my machines, the difference between accelerated encodes and MainConcept are more narrow than for any of the Red Cars. Again, I think that's because the Sample Project benchmark is more influenced by the gpu's ability to handle 1 particular fx than project media. And the Vega64 is a better gpu than it's little Vega-M sibling or the Nvidia 1050ti. With UHD frames, things are closer with the 980x's slower cpu being offset somewhat less by its faster gpu than for fhd frames. If gpu prices ever return to normal, I'll probably add an Nvidia 1660 to it for decoding as I did with the Xeon. Editing performance on the 980x is more brisk than either of these other 2 machines. More similar to the 9900k and Xeon machines. Editing performance seems to be more about the gpu than the cpu. Because the 980x cpu can't hold a candle to either the 9900k or the Xeon.