SD preview of HD footage jerky despite raid & cpu

fausseplanete wrote on 1/25/2009, 1:58 PM
Playback (Good/Full/NoScale) of HD-resolution footage (EX3 mxf) or HDV (m2t) in a same-resolution project works fine. So bandwidth of system (disk etc) must be OK. CPU load is two cores each at 40% approx.

Playback of same footage (Good/Full/NoScale) in a SD-resolution project (implying downscaling in vegas) is jerky. Such downscaling presumably depends on CPU only, since I have read that vegas makes no use of graphic card. But the CPU load is still only at two cores each at 50% approx.

So how come it's jerky? The CPU is only part-loaded and I already established that the hard drive is no bottleneck.

Anyone know the reason why, or has any fix for this (other than lower quality preview / rendering proxies / selective pre-rendering)?

Comments

FrigidNDEditing wrote on 1/25/2009, 2:19 PM
Set your project properties to match media, not destination format, then select render settings of desired delivery formats.

Also set to preview rather than good, as the scaling being done in your preview window is much more intensive at Good or Best.

reasons for cpu usage not being full can be many.

Filters being applied, work that is not able to thread (or receives no benefit from threading), not optimized to utilize multiple cores during playback - just during rendering, and more I'm sure.

but if you aren't applying FX to the footage, my aforementioned changes are your answer.

Dave
John_Cline wrote on 1/25/2009, 3:16 PM
I just posted this in another thread, but it applies to your question about core utilization. It's from a JohnnyRoy post from last April:
--------------------------------------
What people who don't understand how computers do multi-tasking don't realize is that multiple cores can only be utilized when the task has steps that can be done in parallel. Work doesn't magically use all of the cores at 100%.

Think of it this way: You are trying to dig a hole and you have 4 workers standing around a ditch but you only have two shovels (a video shovel and an audio shovel). They can take turns using the shovels 25% of the time each, but you will never get more that 2 workers worth of work done (50% of your total work force). That's what happens in the computer. Unless there are 4 tasks that can be performed simultaneously, you will not use 4 cores at 100%. Rendering to MPEG does a good job of using all of the cores so I assume that there are enough sub-tasks that can be performed in parallel to keep them all busy.
musicvid10 wrote on 1/25/2009, 4:31 PM
Your Subject may have said it best.
RAID and NLE, for whatever reasons, have not always been the best of friends.
I assume from reading others' posts that the same problems that were isolated in the early days, including jerky preview and playback, still persist today at least on some RAID setups.

Sorry I don't have the latest quad / ddr3 / sata / raid machine to test this on, but in addition to the usual advice ( project same as media, preview same as project, simulate device aspect off, etc., etc. ), you might set one of your drives as a conventional non-raid disk and try it that way.
johnmeyer wrote on 1/25/2009, 5:03 PM
Think of it this way: You are trying to dig a hole and you have 4 workers standing around a ditch but you only have two shovels I always preferred: "nine women can't produce a baby in one month."
fldave wrote on 1/25/2009, 5:15 PM
Ha, love the nine women reference!

"Downscaling" says it all. Vegas is having to downscale besides preview all at once.

I always keep my HD projects with HD properties. Going to SD will be just a final render step, not a separate project.

That said, you will need to utilize Dynamic RAM Preview for very short selected regions, or Pre-rendered files for larger regions.
farss wrote on 1/25/2009, 5:24 PM
You simply need to use reasonable RAID controllers. The later mobos do come with dedicated hardware controllers on them. My Gigabyte mobo's RAID controller performs quite nicely and causes only a moderate CPU hit according to HDTach. Playback performance of XDCAM EX's MXF feels slightly better than from my non.
Despite the commonly held belief that data rates from almost any HDD are high enough to feed a vision stream it's rarely that simple when editing, start cutting and mixing and the heads have to travel around both within the one file and between multiple files. Add in a few audio tracks as well and you get the idea.

Certainly editing in a project at native resolution is the way to go. I've been editing a project that a mix of MXF, HDV and some 16:9 SD. The worst playback comes from the SD footage simply because Vegas is rescaling it to HD then having to downscale in back for preview.
Getting all your ducks in a row before you begin editing is the best advice for a smooth experience I can give. That includes all your audio. Get all that at 48KHz to avoid on the fly resampling.
Watch what audio filters you use. Some of the better ones really use a lot of CPU. Don't have scopes running with RT update, that sucks up some CPU. Zoom right out on the T/L. Every time Vegas has to redraw the T/L that takes up CPU and you can easily drop frames.
Needless to say watch out for nasty background processes.

Bob.
fausseplanete wrote on 1/25/2009, 11:01 PM
Thanks all for the tips, though the issue is not so much how to avoid CPU hit as why none of the individual CPUs are being fully hit.

Sounds at first like the "4 diggers two shovels" was in effect, but not convinced, because presumably even the "shovel swapping" i.e. job allocation of which CPU does what at each point in time would be done by CPU also, possibly at the app level. So if the process was CPU-intensive and inherently not parallisable I'd still expect to see at least one of the CPUs max out, even if it was inefficiently job-allocating all the time. But the biggest load on any single CPU (in Task Manager) was only 50%.

It is suspicious that the sum of the two main-loaded CPUs is not far off 100%. Could that be telling a story? Or is it that the Task Manager doesn't tell the whole story i.e. some kind of "stealth processing" is going on (and that one CPU at least is 100% really)?

Can this same effect can be seen on anyone else's system?

I neglected to say the disk was a USB external - a GRAID2. So the system should not even "know" it's a RAID. There was just one clip in the timeline, zoomed into complete view, no cutting / multiple streams etc.

Project at Native resolution I already described as my baseline reference: "Playback ... in a same-resolution project works fine". This is not possible when there are multiple sources of different resolution, including Bob's scenario of SD being upscaled and downscaled (which I guess would also hurt resolution), or where HD is only being used as source into pan & scan.

Set "preview rather than good": I did try that but it made no difference. No filters or scopes were applied.

So I did an experiment of adding some: scopes + color correct + glow + unsharp, also zoom & rotate. I expected to see at least one of the CPUs go up but they still maxed at what looked like 60% for any single CPU. And of course the preview got jerkier.

Continuing the "heavy filter chain" experiment, I put the playback into a loop and played with switching the FX in and out and also the Project Properties. Was able to do that while the loop was still playing! Bizarrely, adding the FX and making the project different to source both actually reduced CPU overall percentage (of the 8 cores total). I think maybe vegas is doing some sneaky cacheing of the loop preview, because as I write this, as it still loops with all FX and different resolution on, the total CPU has dropped to 1%.

Tried all combinations, but the CPU drop to 1% only happens (for my HD footage) when both project is at SD (DV PAL Wide) and the FX are enabled in preview. Disabling FX for preview or making project native causes the CPU to rise again. And vice versa - it is repeatable.

Weird!
farss wrote on 1/26/2009, 1:19 AM
When you loop a section and keep playing it back Vegas slowly fills thePreview RAM cache. That leaves little work for the CPU or anything hence CPU usage will drop.

I have seen your scenario, the cores are not running flat out but things are still slow. Pulling your source from a USB drive might be part of the problemo USB uses more CPU than firewire and way more than SATA.

I'm also uncertain of just what how the task managers meters CPU load. Been a long time since I played with the innards of CPUs but I do recall getting quite different results from software metering and our hardware CPU emulator. That was back in the days of the 68020.
The following from Wikipedia may explain a few things:

"The performance tab shows overall statistics about the systems performance, most notably the overall amount of CPU usage and how much memory is being used. A histogram of recent usage for both of these values are shown. Details about specific areas of memory are also shown.

I've done further reading on this and it's only served to convince me even more of how little I know. It would seem that much of the task of optimising code exceution is left up to the compiler. Optimising code execution over multiple CPUs and cores as well as hyperthreading seems like quite a challenge. I think I should leave further comment to people who unlike me might actually know what they're talking about.

Bob.

fausseplanete wrote on 1/26/2009, 4:27 PM
Bob,

Deep stuff indeed.

Re preview RAM cache: Aha that explains it! But interesting that it only happened when (both together) the project resolution was SD and FX were applied. Under that circumstance, the cache consistently kicked in (CPU dropped to negligible) after three loopings. Disabling FX in either the preview or in the Event FX chain (checkboxes) made the CPU go back up again, as did putting the project resolution up to that of the source media, implying the cache was no longer being used. Complete guess: maybe the cache was invented earlier in vegas's evolution when HD didn't exist and it was thought only necessary when FX were used? Who knows?

Re "USB uses more CPU than firewire..", I'd have thought that would contribute to rather than limit the apparent CPU load (though I use the word "apparent" carefully). It's the limited amount of CPU load that is most mystifying - it appears naively at least that there's plenty lying spare, and not just in the other cores (though in my last paragraph here I guess a possible explanation). As regards the difference in smoothness when previewing at full or reduced scale, it's the same amount of data going in, in each case. So whatever the USB limit is, it's constant. At native resolution, playback is smooth, so I guess that can't be the bottleneck.

Prompted by your highlighting of the "show kernel times" option (which I normally do enable, only partially understanding what it is), I enabled it and repeated the test. Sadly there was no significant degree of missing CPU load revealed by it. Was worth a shot though.

On a totally unrelated computer and application (non video) wot I and others wrote (in C/C++), which uses dynamic memory, it looks like the red line comes up to do things like screen handling and memory reclamation.

The hyperthreading (etc.) challenge you speak of sounds daunting indeed. The author of the venerable VirtualDub gives a glimpse of some of the issues: [http://www.virtualdub.org/blog/pivot/entry.php?id=62]. Way beyond me, though the dangers of deadlocking are familiar at a larger scale.

Simple suggestion: I wonder if the two main cores are 50% simply due to alternate(ish) *instructions* of a single sequential (non parallelised) process (vegas playback etc.) being executed on alternate CPUs or maybe alternate cores of a CPU chip, e.g. done by the chip or OS to spread the temperature load. Each core would have to wait for the other one to finish its step before continuing (like tree-sawing). It would be like having one virtual CPU with 2*50%=100% perhaps. In that case maybe the CPUs are running at full pelt after all. I'm just guessing - and likewise wonder if anyone who actually knows about these things could elucidate, so we can at least get a better feel for the subject.
fldave wrote on 1/26/2009, 7:43 PM
I hate all of my USB drives. I use them totally for off-line backup. Copy your project files to a Raptor or other SATA device and just test to see if you get an improvement.

At times I get better preview with Best/Full as long as no resizing occurs.

Also, the main Vegas engine "use to be" totally based on ancient Video For Windows, VFW. That is why there has been no GPU acceleration, etc. I would bet that the bottleneck is in a non-optimized VFW routine. I said "use to be" because I think V8 may have some new parts trying to get away from that architecture.

So many factors involved here.