Dispelling Rumors - Vegas 8.0 v 8.1 - Compression

John_Cline wrote on 1/24/2009, 2:12 PM
The following is a post by Perrone Ford in the DVi Vegas Forum, he took my "rendertest-hdv,veg" file and tested it in v8.0b and v8.1 in WinXP64 using different codecs. I found the results quite interesting and he gave me permission to post his results here. I decided to start a new thread since it expands on my original thread by adding different codecs. My original Rendertest-HDV thread is HERE, Perrone's original thread at DVi is HERE.

------------------------------------------------
For quite some time now, I've been hearing how using .MOV file types in Vegas is painful. How quicktime is "crippleware" in Vegas, etc. I also knew that I was seeing results in my work that seemed to refute what I was reading.

Last week, I stumbled upon some informal benchmarking on the Sony Creative Suites forum called HDVRender Test. Essentially, the idea was to take a known .veg of generated media, and render it on various machines so people could compare performance numbers. Great idea, so I tried it. Today, I decided to take that one step further. I wanted to use that test to see how various codecs stacked up, and how the two current shipping versions of Vegas stacked up. These are my results:

Machine:

Dell Precision M6300 Laptop
2003 XPPro x64 SP2
Core2 Duo T7250 @ 2.00 GHz
4GB RAM

Vegas Versions:

Vegas 8.1 build 171
Vegas 8.0b build 217

The Codecs: Mpeg-2 HD @ 25 Mbps, Uncompressed, Lagarith, Sony AVC, Avid DNxHD, JPEG2000.

So how do they compare:



Test 1: Stock HDVRender Test

V8.0 MXF 05:46 1440x1080x32 29.970i mpeg-2 HD 25 Mbps(CBR) 48KHz16 bit Stereo 15,810Kb

V8.1 MXF 05:03 1440x1080x32 29.970i mpeg-2 HD 25 Mbps(CBR) 48KHz16 bit Stereo 15,810Kb

** As expected, the 64bit version was significantly faster.


Test 2: Uncompressed AVI HDVRender Test

V8.0 AVI 05:46 1440x1080x24 29.970i Uncompressed 48KHz16 bit Stereo 684,548

V8.1 AVI 05:03 1440x1080x16 29.970i Uncompressed 48KHz16 bit Stereo 456,809

** 64bit render only offered 16 bits per pixel! Speed was faster


Test 3: Lagarith AVI HDVRender Test

V8.0 AVI 05:49 1440x1080x24 29.970i Lagarith 48KHz16 bit Stereo 13,176

V8.1 AVI 05:04 1440x1080x16 29.970i Lagarith 48KHz16 bit Stereo 13,176

** Again 64bit only offers 16 bits per pixel


Test 4: Sony AVC HDVRender Test

V8.0 MP4 02:58 1440x1080x32 29.970i Sony AVC 20 Mbps/512 Kbps 48KHz16 bit Stereo 1,881

V8.1 Would not Render

** This was unexpected as I tried to choose codecs that were in both versions


Test 5: Uncompressed Quicktime HDVRender Test

V8.0 MOV 06:18 1440x1080x32 29.970i Uncompressed 48KHz16 bit Stereo 912,193

V8.1 MOV 03:15 1440x1080x32 29.970i Uncompressed 48KHz16 bit Stereo 912,193

** File size is larger than AVI because it offers 32 bits per pixel. Render speed in 64 bit is nearly twice as fast as 32 bit and much faster than AVI even with increased bit depth.


Test 6: DNxHD 220x 10 bit HDVRender Test

V8.0 MOV 06:01 1440x1080x32 29.970i DNxHD 220x (10bit) 48KHz16 bit Stereo 135,343

V8.1 MOV 01:50 1440x1080x32 29.970i DNxHD 220x (10bit) 48KHz16 bit Stereo 135,343

** Render time is less than 1/3 that of 32bit version and over twice as fast as any AVI.


Test 7: DNxHD 45 8 bit HDVRender Test

V8.0 MOV 06:01 1440x1080x32 29.970i DNxHD 45 48KHz16 bit Stereo 28,542

V8.1 MOV 01:50 1440x1080x32 29.970i DNxHD 45 48KHz16 bit Stereo 28,542

** Same performance gain as 10 bit version


Test 8: JPEG2000 HDVRender Test

V8.0 MOV 06:16 1440x1080x32 29.970i JPEG2000 (32Mbps) 48KHz16 bit Stereo 20,056

V8.1 MOV 03:52 1440x1080x32 29.970i JPEG2000 (32Mbps) 48KHz16 bit Stereo 20,129

** Performance abot 60% better in 64bit version.


In terms of compression:

AVI Files:

Lagarith offers a 52:1 advantage over uncompressed


MOV Files

DNx 220x offers a 6.5:1 advantage
DNx 45 offers a 32:1 advantage
JPEG2k offers a 45:1 advantage



So while this test is hardly scientific, it does seem to offer up some very interesting trends.

1. 64 bit Vegas is CLEARLY faster on the same material using the same codec as 32bit

2. 64 bit Vegas has some significant issues with .AVI files and bit depth

3. The conventional wisdom that somehow Vegas is optimized for .AVI may be true in 32 bit, but the 64bit version blows that out of the water.

So it seems in terms of rendering, that the most speed is available by working with and rendering .MOV file types in 64 bit Vegas. Which is what I've been saying for months now. If the Avid DNxHD codec is used, the render times are faster than anything else around (though I would like to test Cineform) and they can be traded off to Macs or other PCs without cost.

This test took a couple hours to run. I'd be curious to see it replicated on one of the quadcores or dual quadcores out there. Disk speed didn't seem to be a big factor as the render times didn't change much even when writing out uncompressed files versus the far smaller highly compressed intermediate files.

Comments

johnmeyer wrote on 1/24/2009, 2:22 PM
This is a good start, but I think requires a little more rigor to make the results useful. In particular, there are a LOT of settings for each codec, and many of these impact performance. The "quality" slider, as one example, makes a significant difference. As I found out in the brief post a few days ago in the massive HDV Rendertest thread, version 8.1 has different defaults for the MPEG-2 quality slider than does 8.0c and previous versions of Vegas. Thus, comparing the two using the defaults will introduce a bias in favor of 8.1 (when comparing speed) simply because the settings are not the same.

Some codecs use VBR and some use CBR. I think, but am not sure, that with some codecs, like WMV, VBR may take a considerably longer time to encode.

One could simply choose the best settings for each codec, but I don't know what that would accomplish because some codec settings can cause amazingly slow performance when set all the way up (for instance, in audio, try setting the MP3 slider all the way up and measure the performance decrease -- it gets really slow).

Finally, I think if we are going to collectively work on this, we should decide on the goals. For instance. are we trying to compare 8.0c (although he used 8.0b) with 8.1 for multiple codecs? That is something that probably can produce meaningful results and therefore is worth doing.

Or, are we trying to look at how quickly we can achieve encoding with various codecs. That is much, much harder to do because of all the dissimilar settings for each codec, and also because each codec produces such wildly different levels of quality. It is tough to compare apples to apples.

As far as comparing file sizes and levels of compression, I'm afraid that this part of the test is totally meaningless because, as John and others (including me) have often pointed out, this is a function of bitrate, and nothing else. The codec has nothing whatsoever to do with this.

PerroneFord wrote on 1/24/2009, 2:49 PM
John,

In doing the comparison I had to make several choices with the codecs. The choices were as follows.

Uncompressed - Simply chose uncompressed, highest quality, best bit depth available. Reasoning is that anyone who'd take the pain to go to uncompressed would certainly want the best quality available.

Jpeg2k - No settings are available

Avid DNxHD - I only deviated from the HDV template by the bit rate, which was listed for each test.

Lagarith - No bitrate settings available. Used deepest color depth available.

Sony AVC - Used highest Bitrate possible (20 Mbps).


Essentially I only deviated away from the basic render setting for the HDVRender test to produce the best quality output each codec was able to produce. Nothing was done VBR in my test.

I also take some exception to the fact that the file size comparisons are "meaningless". They are of value to a newer video folk. It might help educate people (perhaps less knowledgeable than us) of the various expected file sizes when doing encoding. I have certainly had numerous conversations with newer folks about why they did NOT want to try to encode to uncompressed for uploads to Vimeo and Youtube, despite them having read in some magazine that it offered the best quality.

Take what I did and run with it in any direction you choose. It's utility to me is complete, and I had idle curiosity in seeing how the performance scales to bigger hardware since I have a purchase decision to make in a few weeks time.

Best of luck.
johnmeyer wrote on 1/24/2009, 4:05 PM
I also take some exception to the fact that the file size comparisons are "meaningless".Perhaps I could have worded that better. What I was trying to get across is something that is very commonly misunderstood, namely that the size of a file produced by a given codec depends on one thing, and one thing only: That's it. Nothing else matters, even a little bit.

So, unless there is a bug in the codec, if you set the bitrate to 3,000,000 bps, you will get the identical file size for all codecs. Therefore, comparing file sizes is doing nothing more than comparing what bitrate you happened to use for each codec.

There are bugs in some codecs. For instance, the MainConcept MPEG-2 encoder used in Vegas does not produce exactly the correct size predicted by the bitrate used when encoding single pass VBR. However, if you choose 2-pass VBR, the resulting file size exactly matches what is predicted by a bitrate calculator. This, however, is a bug, not some feature of the codec.

Now, if you can devise some test that somehow compares quality, and you can adjust bitrate until somehow the results look about the same for each codec, than you could determine the efficiency of each codec. For instance, it has been commonly asserted that AVCHD is more efficient than HDV in encoding video, and therefore you can get the same quality -- whatever that means -- from a file of smaller size. Similar claims are often made for DivX and many other MPEG-4 variants. I am not stating that these are false, but only that it is very difficult to scientifically measure what it means, and to prove it.

So, that's why I said, perhaps a little sweepingly, that the comparisons in file size are meaningless.

John_Cline wrote on 1/24/2009, 4:26 PM
John, I'm glad you clarified your statement, I think Perrone was a little put-off by your response. He had not been a member of this forum and I invited him to come join us. He is active in the other two major Vegas forums and I thought we might benefit from his participation.

Regarding video quality measurement, the MSU Compression Project website has just what we're looking for. It requires a working knowledge of AVISYNTH, which I know you have:

http://www.compression.ru/video/quality_measure/video_measurement_tool_en.html

Along with their other amazingly useful, high-quality free plugins, they have a new version of their spatial/temporal video de-noising tool for Virtual Dub that uses the video card GPU to speed up the filter by a factor of 7.

http://www.compression.ru/video/denoising/index_en.html
farss wrote on 1/24/2009, 5:02 PM
There's an additional complication.
The advanced wavelet codecs such as AVC / H.264 / Cineform appear dramatically affected by scene complexity. I've not played with a RED camera but have certainly read reports of the impact of this with that camera. I do use the SI-2K camera that shows how much RAM is being used to buffer uncomp data from the imager which is a good indicator of how hard the CPU is working. Complex scenes with lots of fine detail can fill the buffer quite quickly. If there's motion as well the effect is even more dramatic as I push the frame rate up.

Aside from that I'm still trying to understand the original premise that NLEs are optimised for certain file types or codecs. All NLEs by my understanding have to perform the same set of tasks. Decode all frames required for the current frame, perform calcs determined by FXs, composite and produce an uncompressed frame to pass to the output encoder.
Some FXs require data from previous or forward frames as may the decoding and encoding steps.
Furthermore file types such as AVI and MOV are only containers. Decoding and encoding H.264 like codecs will pose much the same set of challenges regardless of the containing file type.

I should also point out that some advanced codecs have adaptive bitrates and quality. They can 'see' they're running out of time or bandwidth and adapt accordingly.

i hope this doesn't sound too harsh but it seems to me a number of good people have wasted a goodly amount of time after being sucked in by arguments posed by people who simply do not or even want to understand the complexities of what they're talking about. On top of that for me at least there's more pressing issues than how fast or optimised Vegas is for anything. I'm more impacted by it simply not being able to read certain containers such as MP4 requiring me and others to go through a slow and laborious rewrapping step. We're also impacted by it's inability to write certain container types that are in common usage.

Bob.
johnmeyer wrote on 1/24/2009, 6:46 PM
John,

Thanks for that link to MSU. I looked at that many years ago, but boy has it come a long way since then. If I get a chance, I'll spend a little time looking at it.
othersteve wrote on 1/25/2009, 8:06 AM
My testing mirrors these results actually. I haven't done nearly as extensive of testing as you have, but here's what I've got:

Dell M1330 Notebook computer
Intel Core 2 Duo 2.4 GHz T8300 Processor (3MB L2 Cache)
4 GB RAM
7,200 RPM Hard Drive
X3100 Intel Integrated Graphics chip

I used some advice that DSE so graciously provided to me to help with my rendering, though I decided I'd also test 32- vs. 64-bit performance. Here's what I did:

I was running Vista 32-bit originally, and I upgraded to Windows 7 64-bit. I'm a computer tech so I'm no stranger to monitoring background activity and CPU usage/memory usage.

I began with 1 minute of Canon AVCHD footage at 30p, 1920x1080 resolution, 24mbps. I rendered to 1280x720 AVCHD Mainconcept, 3.5Kbps average, 30p, and added a short 4-5 second intro/outro with transition and an image watermark to replicate what I will be doing in real world usage, then trimmed the total video length to exactly 1 minute for ease of calculation. Here were my results:

Vista 32-bit, 8.0c: 8m16s
Win7 64-bit, 8.1: 4m23s

I nearly fell out of my chair.

So take that for what it's worth...

Steve
PerroneFord wrote on 1/25/2009, 8:10 AM
How is your playback performance in the timeline? Looks like pre-render is going to become a constant companioin for me.
othersteve wrote on 1/25/2009, 8:25 AM
Still pretty crappy preview performance, Perrone. I'd say it's about the same in 32- vs. 64-bit.

Steve
blink3times wrote on 1/25/2009, 8:28 AM
"How is your playback performance in the timeline? Looks like pre-render is going to become a constant companioin for me."

That's good ole fashion VFW which SCS needs desperately to tear away from.

There is no doubt that 8.1 with its 64 bit technology is of benefit, but let's remember here that 8.1 is only a first draft (so to speak). I think given another version or 2 vegas 64 will be the norm for the Pro version while the 32 bit version may just fade into the consumer end.

Personally, I'm still on 8c because of my plugins and I don't do much AVC work.... but of the little h.264 work I have done..... 8.1 is clearly the choice.

Thanks John BTW..... for running those tests.
johnmeyer wrote on 1/25/2009, 9:34 AM
I just posted 0:59 in the rendertest thread. Playback performance, however, is only marginally better than my older computer which is now in its seventh year. I think my computer (3.2 GHz i7) qualifies as the one of the fastest Vegas computers you can buy, but the timeline performance in some operations is barely faster than my old computer.

Also rendering on 8.0c is faster than on my old single thread, single core, single CPU computer, but the multi-thread stuff is basically not working. I did render tests on 8.0c using 1, 2, 3, and then four threads (I also hacked Vegas to put the thread count at 8). There was no difference AT ALL between 2, 3, and 4, threads, and the CPU usage looked the same. Only 1 thread produced slower results. Thus, I don't think 8.x (or 7.0d, which I also tested) use multi-core correctly.
farss wrote on 1/25/2009, 2:02 PM
"Thus, I don't think 8.x (or 7.0d, which I also tested) use multi-core correctly. "

What should Vegas be doing to show that it is using multicore correctly?

On a slightly different tack. Can someone try this.
Two tracks of HDV from different sources. Cut out bits of track 1 as you would in a multicam edit. Make certain you've got several minutes of video on the T/L.
I find at the cuts playback performance drops. If I have fast cuts between 3 or 4 cameras playback is not good at all and I have to playout the section several times or shift B to really see how accurate my cuts are. I'm not surprised by this nor complaining about it but it does show the difficulty of using synthetic benchmarks to judge what will happen in real world editing.

From another angle. Mac users running FCP seem to say you should have 2GB per core, one I spoke with recently is running 8 cores and 64GB of RAM! Now I'm running 4 cores and only 2GB of RAM. Vegas says nothing about this however when I fire up AE it does bring up a brief message "Insufficient RAM, multiprocessing disabled", don't know what exactly that means and AE works just fine however it's performance rendering is around the same as Vegas's for the same kinds of tasks.

Bob.
John_Cline wrote on 1/25/2009, 3:02 PM
JohnnyRoy posted the following last April that is the most succinct explanation I have ever read on this subject. He said:
-------------------------------
What people who don't understand how computers do multi-tasking don't realize is that multiple cores can only be utilized when the task has steps that can be done in parallel. Work doesn't magically use all of the cores at 100%.

Think of it this way: You are trying to dig a hole and you have 4 workers standing around a ditch but you only have two shovels (a video shovel and an audio shovel). They can take turns using the shovels 25% of the time each, but you will never get more that 2 workers worth of work done (50% of your total work force). That's what happens in the computer. Unless there are 4 tasks that can be performed simultaneously, you will not use 4 cores at 100%. Rendering to MPEG does a good job of using all of the cores so I assume that there are enough sub-tasks that can be performed in parallel to keep them all busy.
farss wrote on 1/25/2009, 3:30 PM
"Rendering to MPEG does a good job of using all of the cores so I assume that there are enough sub-tasks that can be performed in parallel to keep them all busy."

Not that I have a clue as to what the MC mpeg-2 encoder does exactly however each core can be assigned to a macroblock or I guess one quandrant of the frame. nVidia claim on their ION platform one GPU pipeline is assigned per macroblock.

It's MUCH easier to write code and optimise hardware to solve a specific task e.g. BD playback. As has been discussed here for as long as I've been around it all falls apart when you have to deal with multiple streams of vision, hardware acceleration solutions have little to nothing to offer with unlimited tracks.

Bob.
PerroneFord wrote on 1/25/2009, 3:31 PM
Yep, that's what I am seeing. Moving between 8-bit and 32bit float hardly makes a dent. I tried CineformRAW playback today and it was as dismal almost an DNxHD. Holy cow. So its memory renders, or pre-renders for me until I move to 4 or 8 cores and real RAM.

I'll just keep rendering WMV samples to see my work.
johnmeyer wrote on 1/25/2009, 5:47 PM
What should Vegas be doing to show that it is using multicore correctly?I guess what I was expecting was some sort of change when I changed the "maximum number of rendering threads." I now have a computer capable of eight independent threads. If I limit it to one, I would expect a render to take a certain amount of time. When I limit it to two, I would expect that same render to take less time. Perhaps not exactly half, but at least less time. When I limit it to three, I would expect less time again, and so on.

What I didn't expect was to see no difference whatsoever (0.00% difference) between two, three, and four threads. Also, in 8.1 -- where I DO see exactly what I expected and what I just described -- I also see each virtual CPU maxed out. By contrast, in 8.0c and 7.0d, the Windows Task Manager shows the CPU usage bouncing all over the place, and not getting anywhere near 100% for any of the threads.

So, that's what I think Vegas should be doing, but what do I know ...
farss wrote on 1/25/2009, 7:49 PM
"but what do I know ... "

What do ANY of us know?
The best I can say is I see a bewildering forest of factors that are too much to cram into my skull. I do see that my new quad seems to get things done a lot faster than my old P4 of around the same clock speed. On the same PC I don't see Ppro or AE doing things much differently render speed wise either.

Much of this though seems to relate to the nature of the problem. I see on the OC forums that results vary widely between different CPU designs / number of cores with different benchmarks. It also seems to me that unlike certain things we are moving large amounts of data AND doing a lot of calcs as well. This is kind of unique, I can't think of much else that does both those thing, maybe data mining of large databases does.

Bob.
johnmeyer wrote on 1/25/2009, 10:25 PM
Much of this though seems to relate to the nature of the problem.Yes, I think that is a big part of it, as already described in this post and others (the four guys with two shovels ...).

Rendering is clearly the almost perfect fit for multi-core technology because it is a non-real time process that is completely repetitive, and more important, independent, where what goes on in this frame doesn't depend on more than a few frames on either side.

However, there must be some way to be clever about improving playback performance. If that is going to be forever tied to a single thread, and if CPU performance is maxed out for all time (stock clock speeds haven't changed in seven years), then this timeline performance is what we're going to have forever. That is not a thrilling prospect, to say the least.

As I said before, I don't know what can and can't be done. I only know that my new computer with Vista and 8.1 can render at an amazing speed, but the playback with HD files of any sort is still far from good.

If the following system can't get fast timeline playback of HD, without hesitation when scrubbing or with a few simple fX (e.g., color correction), then I don't know what can ...

[edit] after I posted this a few minutes ago, I re-booted into Vista and tried playback on 8.1. I haven't had too much time to play with this, but on my computer, playback speed on 8.1 under Vista Business 64 is significantly better than 8.0c on Win XP Pro 32 bit. With color correction and levels applied, I was able to sustain 29.97 with the display set to Best Full (i.e., 1440x1080 playback). So, I guess I'd better be a little more upbeat ...

[further edit] I switch to 32 bit, and the display got really dark, and the fps went down to 14 fps. I'm glad I don't have a need for this because this must be very painful to deal with.

Poly X5800A i7 ATX MB SAS,6DDR3,2GLAN,1394,eSATA
Intel i7-965 Extreme CPU 3.2GHz 6.4GT/s 8M 130W
6GB DDR3 1333 PC3 Kit (3x2GB) Memory
Platinum 10-Bay MidTower Black Case(PC60B-Plus2)
700W Quiet PFC SLI Power Supply -High Efficiency
Seagate 1TB SATA-II 7200RPM HD 32M Cache 3.5"
Seagate 73GB SAS 15K RPM Cheetah 16MB HD 3.5"; XP
Seagate 146GB SAS 15Krpm Cheetah hard drive 3.5"; Vista
Trayless SATA Removable Rack (takes 1x5.25" bay)
All-In-One Internal Card Reader Black
Lite-On 22X LightScribe DVD+/-RW Dual-Layer IDE
Sound Blaster X-Fi Titanium PCIe 1x
Nvidia GeForce 9800GTX 512M PCIe2.0 16x Graphics
Samsung 22" LCD 2253BW 1680x1050 2ms Black
On-Board Gigabit/100/1000T Ethernet
Dual O/S Installation
Windows Vista Business 64Bit DVD
Windows XP Professional CD+License
Jeff9329 wrote on 1/26/2009, 12:39 PM
For some reason tits on a boar hog keeps popping into my head.