AVI Plug for REndering Using All Cores?

Streamworks Audio wrote on 1/27/2012, 10:47 PM
I am watching my Performance meter while rendering to a Cineform AVI file... and I noticed that each core is only running at about 40-45%. A few more test with other codecs used (while saving to an AVI file) and each time each core was not used to its fullest.

So I rendered a test using both the Sony and Mainconcept AVC codecs, and for these tests each core was used to it's fullest (or very close to). Same if I am to render to Windows Media files.

So is the AVI Plugin for rendering to AVI files not using all cores (4 in my case)? I know it is not the codec, as I did a test render from an AVCHD file to Cineform in NeoScene and during that transcode all cores were used. Unless the codec used by NeoScene and the one in Sony are not the same?

Comments

PeterDuke wrote on 1/28/2012, 1:37 AM
The number of cores used is dependent on how the software was written.

The more intensive the coding algorithm, the more CPU usage dominates the total coding process and so you will see a higher percentage. For less intensive coding, things like disk access become more important. The less intensive the coding, the bigger the files will be and so more disk access is involved.

There is nothing particularly significant about AVI. It is just a wrapper. Cineform is deliberately written to be easy to process. AVC is a more complex coding algorithm and so you will see more CPU activity.
Streamworks Audio wrote on 1/28/2012, 3:13 AM
This I understand of course. But what I noticed is if I am to transcode an AVCHD file from my camera in NeoScene, watching the resource meter, all four cores are being used up to 90 to 95% - however rendering in Vegas yields each core running about 40-45%. I would think there is a bottleneck somewhere... but if so then why was NeoScene able to use all cores more than Vegas. Then again a quick test in VirtualDub also had only 40-45% being used.

Cineform is a fast codec, and I know AVI is just a container... but Vegas does use a 'plug-in' to write to AVI files, and I thought that maybe that plug in maybe wasn't using all the cores.

Perhaps Vegas is just so well optimized that CineForm does not need all of the CPU to encode ;-)
PeterDuke wrote on 1/28/2012, 4:27 AM
Neo Scene is now outdated I think, and perhaps it was not written to fully use all the cores you have. Does Neo Scene take twice as long to convert as Vegas?

Edit

On re-reading I am now confused. Your first post seems to contradict your second. Are you converting from AVCHD to Cineform in both cases (using NeoScene versus Vegas), and if so which uses more CPU and which is quicker?
Streamworks Audio wrote on 1/28/2012, 12:41 PM
Well in simple terms...

When rendering a AVCHD clip in NeoScene (Cineform HD Link), all 4 of my cores are running at 100%....

http://bucket.streamworksaudio.com/cineform.jpg

But when I render the same file, to the same destination (i.e same drives are used) in Vegas only 50% of each core is used....

http://bucket.streamworksaudio.com/sony_cineform.jpg

So I am wondering why this is.... I did do the same type of test in Virtual Dub (using AVISynth to open the AVCHD file) and saved it as an AVI file encoded with Cineform... but there as well, only 50% of the cores are used.
PeterDuke wrote on 1/28/2012, 3:30 PM
So does Vegas take twice as long or about the same time?

What version of Vegas?

Is the audio getting converted as well as the video?
Streamworks Audio wrote on 1/28/2012, 3:41 PM
NeoScene is faster. Yes audio is being encoding. The resulting file contains the same format for the streams (Cineform video, PCM Audio).

I am still playing around, and not matter how I slice it (in terms of settings) Vegas will not use any more than 50% CPU (well some peaks at 52%).

I even tried giving the Dynamic RAM Preview 6GB, still the same. Another odd thing I see too is that when rendering with the Dynamic RAM Preview set to 6GB - Vegas will grab this much RAM during render... but after rendering is complete that memory used is not released. Have to close the program to release it.

I guess Vegas is not using all cores.

Oh and that is Vegas 11 Build 521 64bit
NicolSD wrote on 1/28/2012, 3:47 PM
This thread got me wondering about the render times I have been getting rendering the RenderTest2010. If I go for a MainConcept HDV 60-1080, the cores are barely working. If I go for a MainConcept MP4 1080P, everything is working hard and I get a 33% boost in performance.

Actually, I get faster times with the CPU only than with the GPU. On an even more positive note, when I render files that take advantage of the GPU, render times are usually 25% faster with the GPU than with the CPU alone.

Please note that my CPU is no slouch: it's an I7-3930K running at 4.7 GHz.. As for the GPU, it's a GTX570 at standard clock speed. I picked it for the memory rather than the speed.
ritsmer wrote on 1/28/2012, 3:52 PM
Often you might improve the CPU usage to nearly 100% by adjusting Number of render threads and also Preview RAM.

On my machine (2 x Xeon quad = 8 physical cores) the "best" setting is about 10 render threads and about 600 MB preview RAM.

This is with Vegas 10.0e and rendering from 1080i AVCHD to mpeg-2 1080i at some 25 Mbps.
Other settings may reduce the CPU usage to about 50%on all 8 CPUs and so nearly double the render time.

It seems that the "peak" values vary depending on the Vegas version - and media in and out.

For Vegas 11/511 with GPU assisted rendering the optimum values are some 2-3 threads and about 60-70 MB RAM - this again on my machine - and might vary for other hardware.

When rendering to- or reading from- uncompressed formats the bit-rates are so high that the disk speed become the bottleneck - and you can not get the CPU usage up to 100 %.

Streamworks Audio wrote on 1/28/2012, 3:59 PM
Of course I checked the thread count just in case it was altered to a new default value when installing the last build... but no - the thread number is set to it's correct values. Plus as mentioned, when I encode to Mainconcept AVC all cores run at 100%, same if I render to Windows Media Video.

So this leads me to think that the AVI Plug in:

aviplug.dll
version: 11.0.0.521
path: C:\Program Files\Sony\Vegas Pro 11.0\FileIO Plug-Ins\aviplug\aviplug.dll

has not been optimized to use all cores. But what is odd.... is if it was taking 100% of one core, that should result in 25% of each core being used (as I only have 4 cores), not 50%.
PeterDuke wrote on 1/28/2012, 4:05 PM
Maybe it is written for two threads max.

The load won't necessarily get distributed over all cores equally. I have 4 cores with hyperthreading giving 8 virtual cores and I often see the load distributed unevenly.
musicvid10 wrote on 1/28/2012, 4:06 PM
AVC is bipredictive long-GOP interframe compression. Encoding requires many times the number-crunching power of Cineform AVI, which is intraframe. Pretty much apples and oranges.
Streamworks Audio wrote on 1/28/2012, 4:14 PM
Of course the are.... I am not comparing the codecs but how they are used within Vegas. The AVI plug in (that writes AVI files in Vegas) is not using all cores. Where the Mainconcept plugin....

mcmp4plug2.dll
version: 11.0.0.521
path: C:\Program Files\Sony\Vegas Pro 11.0\FileIO Plug-Ins\mcmp4plug2\mcmp4plug2.dll

... is using all cores

I know Long GOP formats are more CPU intensive. But that does not change the fact that the Cineform (or any AVI encoder that is capable) should be using all cores to their fullest as NeoScene does.
PeterDuke wrote on 1/28/2012, 4:31 PM
I just tried rendering an AVCHD clip to Lagarith, which is AVI, and the CPU usage was about 40%. Most "cores" were also about 40% but core 3 was about 80%.

Lagarith is less compressed than Cineform so would be more I/O bound.
PeterDuke wrote on 1/28/2012, 4:36 PM
And I tried rendering to AVI uncompressed and the CPU usage dropped to 22% while two of the eight "cores" were virtually unused.
PeterDuke wrote on 1/28/2012, 4:38 PM
I wonder what would happen if we rendered AVI to a RAM disk.
Streamworks Audio wrote on 1/28/2012, 4:41 PM
I tried the same...

AVCHD to Lagarith AVI - again CPU is total between 45%-50%, each core is roughly equal in usage.
Streamworks Audio wrote on 1/28/2012, 4:51 PM
I am not to sure it is I/O....

If I render to Cineform from an uncompressed 540p AVI file (which has a bit rate of 45MB/Sec) source - my HDD caps out at roughly 100MB/Sec (which is the max speed for my HDD). And the CPU sits around 20%-25%, meaning that the bottleneck is the HDD, and that the CPU is spends most of it's time waiting for data.

But if I change the source to the same clip but has been encoded to Blackmagic MJPEG (instead of uncompressed) which has a bandwidth rate of 8MB/sec and render again to the same Cineform settings, the HDD caps out at around 35-40MB/sec and the CPU sits at 50%. So in this situation the bottleneck is the CPU, and the HDD is waiting for the CPU to be ready for the data.

But again the CPU is only running at 50%.... if it was running at 100%, I am guessing the bottle neck would still be the CPU but it wouldn't be as bad, and the HDD should be able to send roughly 70-80MB/sec (as 50% CPU pulls data at 35-40MB/sec)
John_Cline wrote on 1/28/2012, 6:22 PM
There was a thread about four years ago that discussed why all the cores of a CPU were not running at 100% during a render. JohnnyRoy posted the following which is as good an explanation of what's happening as I have heard:

What people who don't understand how computers do multi-tasking don't realize is that multiple cores can only be utilized when the task has steps that can be done in parallel. Work doesn't magically use all of the cores at 100%.

You can read the entire thread here:

http://www.sonycreativesoftware.com/forums/ShowMessage.asp?ForumID=4&MessageID=586955
Streamworks Audio wrote on 1/28/2012, 6:37 PM
OK that is an interesting read.....

But there still is the discrepancy of how NeoScene can render (with the same codec) using all 4 CPUs at 100% while Vegas only does 50%.

The difference to me is that the AVI Plug (which again is responsible for accessing VFW codecs and writing to AVI files) is not optimized to use all threads.

And also to add to that text.... the CPU would be 50% regardless if I include a audio stream or not.

So basically we are still at 'The AVI Plugin is not access all of the CPUs power'
PeterDuke wrote on 1/28/2012, 7:21 PM
Yes, my best guess also is that the AVI wrapping process in Vegas only can use one or two threads. What confounds the issue is that recoding is usually taking place at the same time which can use other threads.
PeterDuke wrote on 1/29/2012, 1:03 AM
Here are some more interesting observations.

I rendered an AVCHD clip to cineform using Neo HD Link and it used about 95% CPU and took 70 secs.

I did the same using Vegas Pro 9c and it used about 30% CPU and it took 114 secs.

So for over 3 times the CPU usage, HD Link was only 1.6 times faster.

(Journalists would twist the figures to say, "For 200% more CPU, HD Link was only 60% faster!")
Streamworks Audio wrote on 1/29/2012, 1:46 PM
Thanks for checking that...

Would be nice if 3 x the CPU meant 3 x the speed... but we did confirm that Vegas (when saving to an AVI) is not using all cores/threads.
John_Cline wrote on 1/29/2012, 4:23 PM
There are simply some tasks which can't be done in parallel using multiple cores. MPEG2 and h.264 encoding/decoding happens to be a task which is particularly well suited to parallel processing considering that it deals with each video frame in small individual blocks. Writing an AVI file is a sequential task.
Streamworks Audio wrote on 1/29/2012, 4:39 PM
Sorry John,

I have to disagree - as we have confirmed that NeoScene is able to write to an AVI using 100% of all cores, something Vegas cannot do. So this is not a question of 'writing to an avi' being multi threaded', but rather that the AVI muxer in Vegas is not multi threaded