New project to use for benchmarks...

Comments

Former user wrote on 12/26/2022, 11:21 PM

@RogerS Is it just with the AI FX that causes crashing?

Because those Intel based AI FX will obviously bias intel CPU's, (8 seconds encode for an Intel, 76 seconds for an AMD) It makes it sort of pointless as an AMD 7950x with Nvidia rtx 4090 will probably always lose to a a much lesser system like an Intel 11700K + Amd 6800, simply because the AMD CPU will never be able to make the gains lost due to lacking the Intel AI acceleration and because Vegas don't offer a different AI framework to run on discreet GPU's, for non Intels.

RogerS wrote on 12/27/2022, 12:00 AM

Yes, it appears to be the AI Fx that causes crashing here.

I do see your point- perhaps we could do two renders with the AI in one but not the other (divide the project up between markers) to be a fairer competition between platforms but still realistically show where AMD or Intel excels.

Not sure if Vegas will never offer a different framework- they've been flexible in other contexts to find ways to leverage hardware capabilities. As of how it works today you are completely right.

Howard-Vigorita wrote on 12/27/2022, 1:23 AM

@Former user Looking more closely at vp19 vrs vp20, vp19 only seems to be sending it's ai to my Intel igpu

Former user wrote on 12/27/2022, 3:55 AM

Howard that is interesting, 8 seconds CPU only for 11th gen. I've seen before that IGPU isn't necessarily faster, but if it's not being used it's an extra resource, but latency can present problems.

Removing AVX512 from 12th and 13th gen Intel look to have hurt performance. This is obviously an old benchmark, I've seen more recent tests, but can't find them now. Howard's 12th gen Intel laptop still seems to be performing worse than it should even if it needs to use AVX2 instead

YoloV4/v5 is a popular AI object/face detector benched on OpenVino

Vincent-Brice wrote on 12/27/2022, 5:19 AM

@Former user Looking more closely at vp19 vrs vp20, vp19 only seems to be sending it's ai to my Intel igpu

Are you saying you can open this project in vp19 or to the way it handles ai in general? Like @Hulk I can't open it in vp19 with a message saying it was created in a newer version of Vegas

Gigabyte Aorus Pro V2 motherboard, AMD Ryzen 9 5950X, Gigabyte AMD RX 6800XT, 64GB Corsair Vengeance 3600MHz, 500GB Corsair Force MP600 Gen 4 M.2 C:drive for windows and programs, 500GB Samsung Sata SSD EVO D:drive for video files, 1TB Samsung Sata SSD EVO E:drive for all other data, Creative Sound Blaster Z sound card, Gigabyte Aorus Waterforce 360 CPU cooler, Thermaltake Core P5 TG open case, Contour ShuttleXpress.

Windows 11 Pro 22H2, Vegas 20 Build 326, AMD driver version 31.0.12027.7000

RogerS wrote on 12/27/2022, 7:29 AM

Howard somehow converted the project for use in 19 and wrote his tests of it above.

Howard-Vigorita wrote on 12/27/2022, 11:32 AM

Yes. Just created all the tracks manually in vp19, then copy and pasted their names, fx, markers, events and keyframes from vp20 running along side. Really did it as a learning experience, never having done adjustment layers or compositing children before. Can't say I fully understand them yet, but at least I can make them all, mute selected tracks to observe the effects, and operate the buttons I had never before bothered to make visible. The design of this thing is absolutely elegant.

The only sticking point for me is the Style Transfer FX has an extra control in vp19 and is more crash-prone due to older versions of Deep Learning and OpenVino. But I also started getting occasional crashes yesterday on my laptop in section 13 running with vp20. But have not gotten one yet on my 6900xt desktop. I do get crashes on the 6900xt doing the Style Transfer ai_fx torture test, but this benchmark is less demanding. Only change I'd suggest is swapping the Woman in the Field section with a pre-rendered clip that looks the same and elsewhere putting in a beefed-up, running Upscale FX which can be more demanding but less crash-prone.

bitman wrote on 12/27/2022, 12:10 PM

I may try it out on my system tomorrow...

APPS: VIDEO: VP 365 (20), VP 19 post (latest build -648), (uninstalled VP 12,13,14,15,16 Suite,17, VP18 post), Vegasaur, a lot of NEWBLUE plugins, Mercalli 6.0, Respeedr, Vasco Da Gamma 15 HDpro XXL, Boris Continuum 2022.5, Davinci Resolve Studio 18, SOUND: RX 10 advanced Audio Editor, Sound Forge Audio Clean Lab 3, Sound Forge Audio Studio 16, Sound Forge Pro 16, Spectral Layers Pro 9, Audacity, FOTO: Zoner, DXO, Luminar, Topaz...

  • OS: Windows 11 Pro 64, version 22H2
  • CPU: i9-12900K with Cooler: Noctua NH-D15s
  • RAM: DDR5 Corsair 64GB (5600-40 Vengeance)
  • Graphics card: ASUS GeForce RTX 3090 TUF OC GAMING (24GB) 
  • Monitor: LG 38 inch ultra-wide (21x9) - Resolution: 3840x1600
  • C-drive: Corsair MP600 PRO XT NVMe SSD 4TB (PCIe Gen. 4)
  • Video drives: Samsung NVMe SSD 2TB (980 pro and 970 EVO plus) each 2TB
  • Mass Data storage & Backup: WD gold 6TB + WD Yellow 4TB
  • MOBO: Gigabyte Z690 AORUS MASTER
  • PSU: Corsair HX1500i, Case: Fractal Design Define 7 (PCGH edition)
  • Misc: Logitech G915, Evoluent Vertical Mouse, shuttlePROv2

 

 

bitman wrote on 12/29/2022, 4:16 AM

@Howard-Vigorita @Former user

For VEGAS Pro Fanmade Ad.veg":

Customized the default Magix AVC 4k (portrait) render templates, changing the height to 2700, width to 2160, frame rate to 24p, and got the following Render times on my i9-12900K /RTX 3090 system (rendering section 13-14):

CPU & QSV around 1m 44s, NVIDEA: 1m 45s - so pretty much for all 3 render types around 1m 45s

Vegas File I/O setting to intel auto QSV, Video to NVIDEA.

I did not see much activity on the NVIDEA GPU. Settings on my PC: no overclocks, not even XMP.

Note: disabling E-cores gave the same result.

 

APPS: VIDEO: VP 365 (20), VP 19 post (latest build -648), (uninstalled VP 12,13,14,15,16 Suite,17, VP18 post), Vegasaur, a lot of NEWBLUE plugins, Mercalli 6.0, Respeedr, Vasco Da Gamma 15 HDpro XXL, Boris Continuum 2022.5, Davinci Resolve Studio 18, SOUND: RX 10 advanced Audio Editor, Sound Forge Audio Clean Lab 3, Sound Forge Audio Studio 16, Sound Forge Pro 16, Spectral Layers Pro 9, Audacity, FOTO: Zoner, DXO, Luminar, Topaz...

  • OS: Windows 11 Pro 64, version 22H2
  • CPU: i9-12900K with Cooler: Noctua NH-D15s
  • RAM: DDR5 Corsair 64GB (5600-40 Vengeance)
  • Graphics card: ASUS GeForce RTX 3090 TUF OC GAMING (24GB) 
  • Monitor: LG 38 inch ultra-wide (21x9) - Resolution: 3840x1600
  • C-drive: Corsair MP600 PRO XT NVMe SSD 4TB (PCIe Gen. 4)
  • Video drives: Samsung NVMe SSD 2TB (980 pro and 970 EVO plus) each 2TB
  • Mass Data storage & Backup: WD gold 6TB + WD Yellow 4TB
  • MOBO: Gigabyte Z690 AORUS MASTER
  • PSU: Corsair HX1500i, Case: Fractal Design Define 7 (PCGH edition)
  • Misc: Logitech G915, Evoluent Vertical Mouse, shuttlePROv2

 

 

John-Ivar wrote on 12/29/2022, 5:03 AM

Kudos for going through the effort of getting it running on V19 @Howard-Vigorita. Also thank you for the compliment!

I usually don't pre-render parts of my projects in order to keep things as flexible as possible, in case I need to change things later down the line. In my mind, an optimized project should run from start to finish in one timeline without crashing. I'd love to make more use of nesting, but in my experience it has been a finicky and risky feature in bigger projects.

As for running the main 'all new VEGAS' project, the software slows down on the initial loading of the Style Transfer effect, but I'm not getting any crashes.

This is the first time I've shared a project for use on other people's builds, and it's been really interesting to hear how it fares. I'm especially surprised at the 'Windows 11 update' project causing crashes for some, as it's basically just a couple of screenshots and native generated media set in motion. I'm not nearly as tech-savvy as many of you in this forum, but it's strange to me that VEGAS would have any trouble at all running that project.

bitman wrote on 12/29/2022, 6:43 AM

@Howard-Vigorita @Former user some update on my render times of section 13-14; I could drastically shorten the render time by disabling the intel IGPU in windows itself, obviously now there are now no QSV times; I observed for:

  • video setting CPU & IO setting off: 1m 27s
  • video setting NVIDEA & IO setting off: 1m 8s (was best time cold CPU; usually higher around 1m 30s / 1m 18s)
  • video setting NVIDEA & IO setting NVIDEA: 1m 14s

Conclusion: CPU heat plays a great role in my system; turning off igpu helps with heat; NVIDEA does not seem to be used a lot in this section. Still a great improvement from 1m 45s with IGPU active.

APPS: VIDEO: VP 365 (20), VP 19 post (latest build -648), (uninstalled VP 12,13,14,15,16 Suite,17, VP18 post), Vegasaur, a lot of NEWBLUE plugins, Mercalli 6.0, Respeedr, Vasco Da Gamma 15 HDpro XXL, Boris Continuum 2022.5, Davinci Resolve Studio 18, SOUND: RX 10 advanced Audio Editor, Sound Forge Audio Clean Lab 3, Sound Forge Audio Studio 16, Sound Forge Pro 16, Spectral Layers Pro 9, Audacity, FOTO: Zoner, DXO, Luminar, Topaz...

  • OS: Windows 11 Pro 64, version 22H2
  • CPU: i9-12900K with Cooler: Noctua NH-D15s
  • RAM: DDR5 Corsair 64GB (5600-40 Vengeance)
  • Graphics card: ASUS GeForce RTX 3090 TUF OC GAMING (24GB) 
  • Monitor: LG 38 inch ultra-wide (21x9) - Resolution: 3840x1600
  • C-drive: Corsair MP600 PRO XT NVMe SSD 4TB (PCIe Gen. 4)
  • Video drives: Samsung NVMe SSD 2TB (980 pro and 970 EVO plus) each 2TB
  • Mass Data storage & Backup: WD gold 6TB + WD Yellow 4TB
  • MOBO: Gigabyte Z690 AORUS MASTER
  • PSU: Corsair HX1500i, Case: Fractal Design Define 7 (PCGH edition)
  • Misc: Logitech G915, Evoluent Vertical Mouse, shuttlePROv2

 

 

Howard-Vigorita wrote on 12/29/2022, 8:24 PM

After doing a Intel graphics driver update for the Intel Arc in my 11900k machine I re-ran timing for the 13-14 section to see if decoding made any difference and it's not happening for me. Does make a difference if I run it once after starting Vegas and then run a 2nd time before exiting. I also re-enabled the igpu in bios and disabled the a770 driver and that changed everything. Timings below for <video prefs gpu>/<io gpu>. All renders done with vp20 build 214 and Win11 set for vp20 to use performance gpu (6900xt).

dnn device=0; 6900xt/Arc a770; 1st run: mc= 0:18,  qsv=0:18, vce=0:21
dnn device=0; 6900xt/Arc a770; 2nd run: mc= 0:10, qsv=0:10, vce=0:10
dnn device=1; 6900xt/Arc a770; 1st run: mc= 0:21, qsv=0:21, vce=0:18
dnn device=1; 6900xt/Arc a770; 2nd run: mc= 0:10, qsv=0:10, vce=0:10

As before, load in gpu charts shifted with the change in dnn device. But a very different picture after I disabled the Arc and enabled the igpu. I also tried setting the video prefs gpu to the Arc and it locked up every time @ 50% where the Style Transfer fx kicks in. ... I'm pretty sure I was able to do that without lockup using the older driver but the timing wasn't very good.

dnn device=0; 6900xt/uhd750; 1st run: mc= 2:33, qsv=2:33, vce=2:33
dnn device=0; 6900xt/uhd750; 2nd run: mc= 2:10, qsv=2:10, vce=2:10
dnn device=1; 6900xt/uhd750; 1st run: mc= 2:33, qsv=2:33, vce=2:33
dnn device=1; 6900xt/uhd750; 2nd run: mc= 2:10, qsv=2:10, vce=2:10

Load stayed on the igpu and timings for all the renders were the same. The OpenVino/Deep Learning apparently locks onto the igpu if it sees it. Next I tried disabling both the igpu and the Arc. This made a difference sending the load entirely to the cpu and was a little quicker than the igpu.

dnn device=0; 6900xt/off; 1st run: mc= 1:54, qsv=na, vce=1:57
dnn device=0; 6900xt/off; 2nd run: mc= 1:31, qsv=na, vce=1:33

Odd that MainConcept ran a little quicker than Amd Vce both times. Lastly I tried setting video prefs gpu to the igpu and it was slower, but did not lock up.

dnn device=0; uhd750/uhd750; 1st run: mc= 2:50, qsv=2:50, vce=2:50
dnn device=0; uhd750/uhd750; 2nd run: mc= 2:41, qsv=2:41, vce=2:41

I did get a few crashes, btw. But only on exiting Vegas once or twice after what looked like a successful run. But was always able to do a do-over with the same timing and no crash on exit. My big take-away was to disable my 11th gen igpu again. And just wait for new Vegas, Deep Learning, and driver updates to improve reliability.

Hulk wrote on 12/29/2022, 10:34 PM

Yes. Just created all the tracks manually in vp19, then copy and pasted their names, fx, markers, events and keyframes from vp20 running along side. Really did it as a learning experience, never having done adjustment layers or compositing children before. Can't say I fully understand them yet, but at least I can make them all, mute selected tracks to observe the effects, and operate the buttons I had never before bothered to make visible. The design of this thing is absolutely elegant.

The only sticking point for me is the Style Transfer FX has an extra control in vp19 and is more crash-prone due to older versions of Deep Learning and OpenVino. But I also started getting occasional crashes yesterday on my laptop in section 13 running with vp20. But have not gotten one yet on my 6900xt desktop. I do get crashes on the 6900xt doing the Style Transfer ai_fx torture test, but this benchmark is less demanding. Only change I'd suggest is swapping the Woman in the Field section with a pre-rendered clip that looks the same and elsewhere putting in a beefed-up, running Upscale FX which can be more demanding but less crash-prone.

Would you be willing to share you VP19 project with those of us who don't have VP20 so we can test out this project on our rigs?

Howard-Vigorita wrote on 12/29/2022, 11:34 PM

@Hulk Be happy to share it if it's ok with @John-Ivar. I'll pm him with a link in case he wants to look it over to make sure I didn't mess it up.

John-Ivar wrote on 12/30/2022, 1:49 AM

Feel free to share, @Howard-Vigorita 👍👍

Vincent-Brice wrote on 12/30/2022, 3:45 AM

@Howard-Vigorita I was glad to see you recorded 1st run, 2nd run. In @RogerS Benchmark project I always wondered what we should record. I did several runs and just recorded the fastest, thinking everyone else was probably doing the same. But the difference between first run and second run on this project is huge.

My all AMD system results are

Full project

1st run: AMD VCE=3:27, MC=4:01

2nd run: AMD VCE= 3:01, MC=3:38

Section 13-14

1st run: AMD VCE=1:28.62, MC=1:30.79

2nd run: AMD VCE=1:00.14, MC=1:02.60

What causes the difference in times? (I notice RAM builds during first run and is already there for the second).

So, the question is, what figures should I have recorded in your Benchmark project Roger, first or second run?

Last changed by Vincent-Brice on 12/30/2022, 3:57 AM, changed a total of 1 times.

Gigabyte Aorus Pro V2 motherboard, AMD Ryzen 9 5950X, Gigabyte AMD RX 6800XT, 64GB Corsair Vengeance 3600MHz, 500GB Corsair Force MP600 Gen 4 M.2 C:drive for windows and programs, 500GB Samsung Sata SSD EVO D:drive for video files, 1TB Samsung Sata SSD EVO E:drive for all other data, Creative Sound Blaster Z sound card, Gigabyte Aorus Waterforce 360 CPU cooler, Thermaltake Core P5 TG open case, Contour ShuttleXpress.

Windows 11 Pro 22H2, Vegas 20 Build 326, AMD driver version 31.0.12027.7000

RogerS wrote on 12/30/2022, 3:57 AM

For the benchmark I manage I'd record what is representative of your system. On my laptop which heavily throttles I use the first run and let it cool before another. That's how I do renders on it as well.
On my systems repeat renders are within 1s or so, even between versions of Vegas, so it doesn't matter. How much does it vary on yours?

I'm not sure why there's such a difference on this project or why the second would be faster than the first. I'm unable to complete any renders in 20 so can't really say more until a future build of 20 comes out that hopefully has more stable AI support with my Intel/NVIDIA systems.

John-Ivar wrote on 12/30/2022, 6:57 AM

@RogerS I've updated the "All New VEGAS" project. It now uses a Windows default font. The Style Transfer effect is disabled by default, in order to ensure users with differing builds or incompatible drivers or hardware don't crash on launch. Maybe this somehow fixes something for you to complete a render, Roger?

I've also added a region marker for the "AI-assisted" section you've all been testing in this thread, @bitman @Former user @Musicvid @Hulk @Vincent-Brice @fr0sty (thank you for the idea @Howard-Vigorita)

Howard-Vigorita wrote on 12/30/2022, 2:38 PM

@John-Ivar Thanks so much. @Hulk Here you go... it's only the Vegas project rebuilt with vp19 b648 reflecting the text font changes, unchecking Style Transfer event FX now on track #11, and a couple of clips renamed since the original... you should still get the vp20 version from the link above for all the media that goes with it.

https://drive.google.com/file/d/1ebcDpznDzozPiduAEnf_TEnTQTLQv_Ic/view?usp=share_link

VP19 CAUTION: vp19 b648 seems to get stuck in memory on all my systems after using any ai-fx (like Colorize, Style Transfer, or Upscale). I either need to run it under an auto-termination script (like the PowerShell script in my signature) or kill it with Task Manager.

Howard-Vigorita wrote on 12/30/2022, 3:19 PM

@Howard-Vigorita I was glad to see you recorded 1st run, 2nd run. In @RogerS Benchmark project I always wondered what we should record. I did several runs and just recorded the fastest, thinking everyone else was probably doing the same. But the difference between first run and second run on this project is huge.

What causes the difference in times? (I notice RAM builds during first run and is already there for the second).

So, the question is, what figures should I have recorded in your Benchmark project Roger, first or second run?

I think since that section is so short that Vegas can cache the decoding and/or possibly some of the ai setup, making the 2nd run go faster. I never see that much variance in Sample Project or Red Car... I usually run them twice and report the quickest and notice that without enough of a cool-down delay, the 2nd run will take longer. I happened to have the lid off my desktop and wasn't expecting to see any time improvement on 2nd run here after the way it made my gpu and cpu pumps churn. I repeated it a number of times before I decided it wasn't me.

RogerS wrote on 12/31/2022, 1:33 AM

Thanks for this.

Here are some tests on my desktop

(AI is off)
First NVENC run 2:16
then Mainconcept at 2:08
then 2nd NVENC run at 1:35.
Restart Vegas and another NVENC run: 1:41.
Restart and Mainconcept run: 2:14
Restart, break and Mainconcept run: 2:16
Restart, break and NVENC run: 1:40
Back to back NVIDIA run: 1:40
Back to back Mainconcept: 2:11
NVIDIA run in background: crash
NVIDIA run in background: 1:39

First run must have been Mainconcept based on the time- maybe user error? I'll restart my computer and test again.

At the same bitrate settings the Mainconcept render is 170MB vs 180MB or so for NVENC. Not sure if it matters much.

RogerS wrote on 12/31/2022, 2:11 AM

I did a system restart and though I was doing a NVENC render but the time and file size indicated it was MC. Did a NVENC render again and got the normal time.

So watch the settings carefully as I'm not sure how I messed up NVENC and Mainconcept twice.

Render times don't see to change from run to run here.

RogerS wrote on 12/31/2022, 3:00 AM

@John-Ivar @Howard-Vigorita

I made a draft sample data collection form with instructions for this project.

I'd like to keep it simple so we get as many comparable results as possible. At the moment I didn't include a lower res option and just MagixAVC. For the last benchmark, 95% of responses were for MagixAVC.

In the last benchmark we also had a playback test between 2 markers. Are there two markers here that would be a useful test for systems (not too easy and a relatively steady framerate would be best).

Howard-Vigorita wrote on 12/31/2022, 12:06 PM

The 2 render templates match what I've been using for Nvidia and MainConcept. Here's my custom Amd Vce constructed the same way by editing the Magix/AVC default and customizing frame size:

Looks like the data fields do not yet include Section #13 with Style Transfer FX enabled. Since enabling this might dominate the render time or lock-up Vegas, probably best to run the whole bench with that disabled (as-distributed) and then have a field for just that section with the Style Transfer FX enabled. Users could enter crash or lockup here if they cannot get it to run enabled.

You might also generalize a bit to select more than just Magix/AVC... allot of folks like to use Voukoder instead. Also suggest you get away from calling CPU a render mode since there is no selection named that in Vegas. Instead use MainConcept or MC as a Magix/AVC button and a field for Other and/or more radios for some of the popular Voukoder options like h.264, h.265, Nvenc/h.264, Qsv/h.264. Qsv/hevc, etc.

In addition to vp20, a button for vp19 would also be helpful to provide comparative data for those users considering an upgrade.