Number of cameras, and playback speed

megabit wrote on 11/24/2011, 1:39 AM

While fighting the weird behavior of RED RAW files in my multi-camera project (http://www.sonycreativesoftware.com/forums/ShowMessage.asp?MessageID=787779&Replies=5)

- I found out that it's NOT the track of RAW high-res R3D files that - when included in my 10-camera multi-track -slows down playback dramatically (from 25 to 5 fps).

There seems to exist a limit of 8 tracks (no matter which format), below which I'm getting full 25 fps even at Best/Full. Add a single more track - even a lower res - and the 9 or 10-camera multi-track will slow down and crawl at a pathetic 5 fps, even at Draft/Auto on my new fast HW!

Judging from the GPU load monitor, my GTX 580 is hardly used at all when the number of tracks in multi-camera exceeds 8. Below 8, it's working happily at some 60% load (note there is still plenty of horsepower headroom), giving excellent playback speed. Above 8 (no matter 9 or 10) - it's not used at all!

Anyone noticed that?

Piotr

PS I've just filed a Support ticket on this (and to think that I've just spent a lot of money on upgrading my HW with multi-camera projects in mind... argh!!!)

Oh, and it isn't a HDD bottleneck at play - I have a RAID 0 capable of over 300 MB/s transfers. Unlike GPU, CPU is taxed at almost 100% during playback of such a multi-camera track - so the data is there; it's just that the GPU acceleration is not kicking in at all.

Comments

farss wrote on 11/24/2011, 5:48 AM

Me thinks you've hit a disk bottleneck.

1) A lot of mobo RAID implementations use the CPU to do all the calcs.

2) 300 MB/s in your scenario can be a meaningless figure. You're trying to get X amount data from Y number of files, quite different problem from getting X amount of data from 1 file. In the former it's not just the data rate but the rotational latency and seek time that starts to kill performance. That's why large corporate application servers and serious editing systems use large arrays of small capcity SAS drives that run at 15K connected via fibrechannel and the like. Those kinds of disk arrays are very expensive. You also need to ensure your mobo is up to it as well.

Bob.

megabit wrote on 11/24/2011, 6:23 AM

Yeah, maybe - my next move will be to re-arrange the project, so that the source files are split (roughly 50:50) between the TWO arrays I have....

But before I do, I'll give SCS Support some time to address my suspicion there is a hard-coded limit to how many simultaneous sources can be accelerated with CUDA/OCL. 'Cause you see Bob - watching the Performance Monitor during playback, I can see HDD activity of up to 40-ish MBps, which is well within my array's capabilities.

But you may be right - I'll report back

Piotr

paul_w wrote on 11/24/2011, 6:57 AM

Well for what its worth, I once fixed my multicam projects by using separate drives to span the clips out and preview from there. It helped a lot. Getting major problems with v11 though :( thats another matter.

Paul.

megabit wrote on 11/24/2011, 7:19 AM

Well, spanning my sources over 2 arrays didn't help at all - still the same threshold of 8 tracks (full fps playback); add 1 - the playback slows down to 5 fps, with no GPU activity at all.

So I definitely hit something - as Bob put it - but not HDD bottleneck, I guess. When watching the Performance Monitor, the only resource that seems to have no headroom is RAM: hard faults are constantly at the max!

Since I have quite fast RAM, I guess what I did hit against is some lack of optimization in Vegas, or even - a hard coded limit of tracks/data for the GPU acceleration to be engaged....

Piotr

megabit wrote on 11/25/2011, 3:05 PM

Guys,

I'm still having the problem with my 10-camera project, and SCS support hasn't answered yet. So - as many times before - I'm kindly asking those of you guys who:

- have a similar hardware (GTX 580, i7-2600K, 350 MBps-capable disk system)
- have a little time over the weekend
- are willing to help :)

- to please set up a 10 HD track project, combine ALL the tracks into a multi-camera track, and just check how it plays back when compared to just 8 tracks combined...

I'd appreciate your feedback very much indeed, as I've a very tight deadline, so at least I'd know whether or not it's worthwhile trying different hardware and software settings to speed the playback... As I said, it seems 8 is the maximum number of simultaneous tracks for the CUDA/OpenCL acceleration to kick in - it might be some system bottleneck (in which case it might be possible to eliminate it), or VP 11 "feature (in which case of course I can't do much about it, and have to find some work-around in my workflow)..

TIA

Piotr

paul_w wrote on 11/25/2011, 3:33 PM

Confirmed here too.
Just tried 8 HD tracks, made into a multicam track, playback was 25fps in preview FULL mode. GPU on.
Went to 9 HD tracks, playback drops to 2-3 fps.
Also tried 10 HD tracks, still bad, 2 fps.

Dramatic!

My hd tracks were 1280x720 MOVs from a small canon camera. So not the full HD but effect was the same.

V11 build 425.

really supprised it didnt crash, could be because the clips were quite short, like 30 seconds.

Paul.

farss wrote on 11/25/2011, 3:35 PM

In the absense of anyone being able to offer any concrete results I'd suggest a rethink of your workflow might be in order. I'd point also point out that other NLEs have hard codec limits to multicam so getting 8 cameras in Vegas is quite a bonus but obviously plenty in the business can live with a lot less. Of course there is plenty of content on air that's used a massive number of cameras but that's from using vision switchers where scaling the number of cameras is a piece of cake.

Pick you main cameras, the ones that you'll use to tell the story. Cut with them, get the story told then look at your other cameras for the icing on the cake snippets and cut them in. I don't even use "multicam". I create a project with all the cameras synced, make a copy, ditch the "cameras" that aren't part of the guts of the story, get the story told and if time etc permits go back to the all-in-one project to look for bits that'll work to enhance the story. More often than not my other "cameras" are shots from rehearsals.

Bob.

megabit wrote on 11/25/2011, 3:39 PM

Thanks so much Paul for confirming this - please do file a case with SCS!

I hope they will eliminate this in the nearest update, but it will be too late for my current project anyway :( All my 9 tracks are 1080/25p, and the 10th one is the RED RAW format - hundreds of clips as the concert was almost 2 hours long.

Piotr

paul_w wrote on 11/25/2011, 3:43 PM

Certainly will, right now. Already got two others going! They're gonna hate me :(

Paul.

megabit wrote on 11/25/2011, 3:46 PM

Bob,

Now that I know further effort to find and eliminate a bottleneck would be futile (thanks to Paul), I'm going to lose no more time and devise some sort of a work-around; since 8-camera track plays back at full speed and Best/Full preview quality (thanks to my GTX 580), I'll just "tell the story" with 8 cameras (excluding the 2 wide and static ones, one of which luckily is the RED), and render it out. In the second stage, I'll cut between the 3... Does it sound like a plan, do you think?

Piotr

paul_w wrote on 11/25/2011, 3:55 PM

Submitted.

Paul.

megabit wrote on 11/25/2011, 3:58 PM

"really supprised it didnt crash, could be because the clips were quite short, like 30 seconds."

Yeah - I must admit one thing in favor of VP11: on my new system (only build a couple of days ago), it NEVER EVER crashed, not even froze!

And I've been doing crazy things to it while trying to find a solution to this problem - like loading another project (how big they are I already gave you an idea of) while one was playing back in multi-camera edit mode, and not even stopping the playback! Switching constantly from separate to combined tracks, undoing changes - all without stopping playback...

And not a single crash. I know - you don't believe me, but it's true :)

Piotr

paul_w wrote on 11/25/2011, 4:03 PM

I believe you. But i just wish my version here was so stable. Im getting crashes just playing multicams on the timeline whenever it feels like it. Nevermind.. waiting to hear back...

Bobs advice is good though, telling the story with the right cams and not just trying to get everything on the TL. Agreed.

Paul.

farss wrote on 11/25/2011, 4:11 PM

"Does it sound like a plan, do you think?"

Yes but I don't know your "story" of course but in general the wide carries most of the story. With just a locked off wide shot of a whole concert it is going to become a bit boring but then again it is all about the music and the performance, the vision is just there to fill the screen.

Bob.

megabit wrote on 11/25/2011, 4:23 PM

You're absolutely right - but all the other (non-fixed) cams have plenty of useless moments in their coverage (when the operator would rapidly pan or zoom to find the most interesting subject/framing during the performance), so the initial cut will also serve the purpose of getting rid of them, and leaving only the best takes in the rendered film. Only then - using that "technically clean" material along with the 2 static cameras - will I be able to tell the story....

Frankly, I'm trying to sort that out in my head while typing this - it's only been an hour since Paul confirmed the "limit of 8" - so my concept is subject to change of course. One of the reasons I'm planning to only use the wide fixed cameras in the second stage is that - one of them offering this huge resolution - I'd like to do some pan and scan around it with the "tiny" 1920x1080 window...

Will see how it works.

Piotr

megabit wrote on 11/25/2011, 4:41 PM

A couple of interesting observations on how Vegas is performing when the number of tracks exceeds 8, and the playback rate drops to 4-5 fps... Even though only some 5 out of my 16GB of RAM is used, the Performance Monitor shows heavy trashing (permanently high number of memory hard faults).

Right now I'm running a Moldflow analysis which requires very large sets of data to be wrangled between RAM and HDD (Task Manager's 'Commit' shows 17/39 GB at the moment), so the disk is working permanently - and yet the number of hard faults in RAM is zero, or some 5-6 max and only now and then!

Well, I guess the above comparisons says a lot about the quality of programming ...

Piotr

farss wrote on 11/25/2011, 4:58 PM

"Well, I guess the above comparisons says a lot about the quality of programming ..."

What is happening is explained here.

The term "Hard Fault" can be a bit misleading although handling them uses up a lot of resources and the fewer of them the better things will hum along.

Bob.

paul_w wrote on 11/25/2011, 5:11 PM

Thats interesting stuff Bob, reading that unhandled page faults can "cause the application to terminate" due to the default handling of the operating system. Hmm, sounds familiar...

Paul.

megabit wrote on 12/9/2011, 5:50 AM

Paul,

Have you ever got any response from SCS Support on this issue?

Piotr

paul_w wrote on 12/9/2011, 6:52 AM

No response, still waiting. Did you file a case too?

Paul

megabit wrote on 12/9/2011, 7:13 AM

Yes I did, and no answer, either :(

Piotr

paul_w wrote on 12/16/2011, 6:54 PM

Latest: They replied to the support request and we now have a new build of Vegas. The email seems to suggest this new build 511 should be an answer?
But after testing - it is exactly the same. Sorry. Still the 8 cam limit.
I tried.

Paul.

megabit wrote on 2/27/2012, 8:30 AM

Just tu update those interested that SCS has finally confirmed the issue (the limit of 8 cameras for MC GPU acceleration), and promised to fix it in the coming releases !

Looks like SCS is really trying to help us - the loyal customers - do our job :)

Piotr

paul_w wrote on 2/27/2012, 10:40 AM

wow, really? thats excellent, had given up all hope of this being fixed.

good on ya SCS.

Paul.