@John-Ivar Hi John, I wanted to ask if there's a region of this project you think would be a good challenge to systems for playback framerate. We had this in the VP 16 benchmark as well. It's preferable that it not easily play back at full FPS and also be representative of the type of Fx an editor would use in a normal project. I'll test out some regions but if you have recommendations let me know.

Then I can finalize this and start to encourage submissions.

Cheers for the project; it's really well-done! I'm experimenting with it a bit to see if it might make for a good playback or general encoding test with my benchmarking. But, since it uses CPU and GPU to good effect, it might be best for system comparisons rather than direct CPU or GPU comparisons.

Nonetheless, to add some commentary to things mentioned earlier: the AI FX are completely CPU-bound, so any GPU usage seen during those tests is the GPU likely doing something superfluous. I've found that the AI FX are really broken on Intel's 12th and 13th-gen CPUs, due to their big / little core design, which would impact results here if Style Transfer is left enabled.

I've been doing some CPU testing this week, so here's a clear-cut example of the problem:

Style Transfer AVC Encode Time

AMD 7950X: 74s
AMD 7900X: 92s
AMD 7700X: 119s
AMD 7600X: 148s
Intel i5-12600K: 333s
Intel i9-13900K: 354s
Intel i5-13600K: 364s
Intel i9-12900K: 373s

Colorize AVC Encode Time

AMD 7950X: 97s
AMD 7900X: 120s
AMD 7700X: 156s
AMD 7600X: 196s
Intel i9-12900K: 435s
Intel i5-12600K: 473s
Intel i9-13900K: 484s
Intel i5-13600K: 513s

These results are the best of seven runs (in a row).

I plan to mention this to the devs soon. I've been doing more experimentation and want to make sure I don't run into something else that might also need mentioning.

Thanks for testing with a range of systems! Agreed on it being a good test of system performance.

I don't think AI is completely CPU bound, it depends on the system and build of VP 20 (try pre- 12th gen Intels).

For that reason I don't want to do anything with this benchmark and AI until the next VP 20 build comes out and see if that further changes performance and compatibility. Based on another thread here it sounds like it's imminent (fingers crossed!)

I think the region between 1 and 2 is long enough to do a preview FPS test so I added that. Once the next build of 20 comes out I'll encourage submissions.

Sounds good @RogerS. As for which regions are best for performance testing I think you have a better idea than me at this point. You probably have some idea through your testing.

The AI segment has provided some interesting results here in this thread, but might not be representative for what a 'normal' project would include? Although it is a highlighted feature for VEGAS, so I guess stress testing it on different builds is beneficial for the feature in the long run.

I'm sure it'll be fine either way.

Thanks John. The first region is pretty hard to playback so it should help show differences between systems for a while to come. I didn't see huge variation from start to finish so it should be feasible to estimate an average unlike other sections. It does seem sensitive to dynamic ram preview usage so I will ask people to zero that out.

The AI colorization is really all over the place in my testing. Each build of 20 seems different and even run to run I get variation on the order of 3x on the same system with the same settings. At times Vegas just seems to largely stop with the render countdown freezing and then picking up again 10 or 30 seconds later. Hard to say what this test shows at present but I think you're right it will be interesting to watch how Vegas evolves with it and different hardware going forward.

I have to agree that Colorize and Style Transfer are kind of a special effect not likely to be used in everyday projects. Of the two, I find Style Transfer the least demanding. My only issue is that Style Transfer often crashes my systems with Intel igpus unless I disable them or force ai processing to the cpu. Upscale, on the other hand, is something I find myself going to in more projects. It's totally stable but is incredibly demanding on scales above 1.100x. So much so that I find I need to frame my shooting better so that I only need to use it sparingly and at low scaling. Looking forward to thoroughly benching ai-fx on the upcoming release.

This thread is about to get really interesting really soon... I'm looking forward to seeing the results.

Now that VP 20 update 2 is out, download the project and submit your results here!
https://forms.gle/1Sb1R61mzRNF44KU9

The new vp20 update (build 326) now runs my old ai-fx test and does not crash with Style Transfer enabled! Only problem is the test now runs so fast on my laptop, I had to lengthen it for a more meaningful spread of times running the different fx... will upload and post timing results for the longer version in it's own thread.

@John-Ivar Could you clarify the settings for the AI style transfer? On my laptop it's still acting strangely with no effect showing up. While the preset is "mosaic" it comes up set to Udnie which doesn't do anything here. If I change it to Mosaic preset it works though original seems to fine a detail for what your original render looks like. Should it be low or medium details?

Submitted results for my laptop... but i incorrectly labeled it nvenc, when it was mainconcept. Oddly, I tried to go back and do a Nvenc render after that, and it was taking more than 2x as long as the CPU render!

Hi Frosty, no problem- I noticed that and was going to ask you. I fixed it. Weird

Maybe try a second NVENC render with the AI Fx turned off and see if it works. I've had odd hiccups with this project where it's slower than it should be at times.

That was the thing, I did the render once with it off (it's off by default, right?), then enabled AI, rendered that out, reported that time, and then when I disabled it again, that was when nvenc was running so slow. Maybe something was still slowing it down from AI being previously enabled. I'll give it another go later.

Hi @RogerS, I posted my times for the new benchmark but then realised I didn't do the AI enabled bit correctly, I recorded the entire project, not just section 13-14. My time for just this section with Style Transfer enabled was VCE 0:07.35 and MC 0:07:46. But then this seems stupidly low, so what am I doing wrong now?

Hi Vincent, thanks for posting. Understood about the AI mistake. Can you doublecheck that the rendered file actually shows the AI effect? My laptop "rendered" the AI file in 20 seconds, which is far faster than my desktop. When I opened the file the "AI" part was just black so I threw out that time.

I asked John to confirm the settings and may update the instructions.

Just re did it, again 7 odd seconds with AI showing in resulting video:

Edit: I just put the above video clip, maybe someone could check I'm doing it right. (I couldn't even put that in the right place in the comment 😏)

@RogerS ... I screwed up putting the render times in. Neglected to use 2 digits for the minutes field. My apologies.

Hi Howard, no problem at all. I check the results from time to time for data cleanup afterwards. It's all sorted happily now.

@Vincent-Brice Thanks for that video! It looks like you did everything right and your system processes the AI Fx almost as fast as a normal Fx. I updated your AI times accordingly.

The AI Fx aren't working as expected on Intel / NVIDIA systems. I hope we can get some more AMD results (both CPU and GPU) to see what's going on in Vegas.

@RogerS, for the playback frame rate, my preview was not set to Best/Full. Please update to 3.0 for my entries.

Thanks Illusion! I also moved the Voukoder result to comment to keep the data comparable (interesting there wasn't much difference).