Multi-generation DV render test

Chienworks wrote on 12/2/2003, 12:38 PM
I finally decided to try one of these myself since it seems to have been a topic of some interest lately. I've uploaded the results here: http://www.vegasusers.com/testbench/files/generations/
Be sure to check out the readme.txt file for a description of what it all means.

I started out with the still image in the 0-start_image.png file. This image is lossless 4:4:4 created from various generated media and scanned 4:4:4 images. Note in particular the crisp edges and even color in the test bars in the upper left corner.

I rendered this image to a standard NTSC DV .avi file in Vegas 4.0d using the included SONY DV codec. The result is gen-00.png which is now 4:1:1 as per the DV spec. Note how the edges seem to spill into each other. Also the edges between the red sweater and black vest are very blocky.

diff-start-00.png shows the difference between the original image and the first DV version. The differences are slight, but noticeable.

This .avi file was then rendered to a new .avi file. Re-rendering, rather than simple copying, was forced by overlaying the new generation number on the frame. This result is shown in gen-01.png. There is very little difference between generation 00 and generation 01.

diff-00-01.png shows the difference between generation 00 and generation 01. There is almost no visible difference.

The process was continued, re-rendering each subsequent generation from the previous one, through 100 generations. The last generation is shown in gen-99.png. Even this far out, the image is still nearly identical to the first DV generation.

diff-00-99.png shows the difference between generation 00 and generation 99. The difference is so slight as to be almost unnoticeable.

It would appear that the major quality hit is the conversion from 4:4:4 colorspace to 4:1:1. Even this hit is minor and limited mostly to areas of extreme color contrast. The colorbars were affected the worst and the red/black of the sweater and vest somewhat affected. Skin tones, the leaves, the cookie, and other less than saturated colors were affected very little at all. Overall the color balance of the 100th generation was still very faithful to the original and the detail only slightly softer.

generations.avi is a short video showing the progression through each re-rendering generation. Place this video on Vegas' timeline and you can use the left & right arrows to step through them.

Comments

Liam_Vegas wrote on 12/2/2003, 12:55 PM
Thanks very much for doing this. That does (for me anyway) clear up any concerns I had about the generation loss.
mjroddy wrote on 12/2/2003, 1:18 PM
While I didn't understand the pics that started with diff-, I appreciate all the effort you put into this and your sharing your amazing results.
"diff-" pics looked like big black graphics with big numbers in the lower left corner. I didn't see befor/after or even full pics.
But reading your results taught me a lot.
Thanks again.

mjr
Chienworks wrote on 12/2/2003, 1:32 PM
mrjoddy, a "difference" picture is a combination of two pictures, with one subtracted from the other (more or less). Wherever the difference picture is black, the two original pictures are absolutely identical. (Think of black as "zero"; two numbers which are identical have a difference of 0.) Wherever it's not black, the brightness of that area is proportional to how different the two pictures are. Since the diff-00-01 picture is almost entirely black (except for the weird numerals, which were of course different since one was a "0" and the other was a "1") this shows that generations 00 and 01 are almost identical.

Even in the worst case which was diff-start-00, the areas of most difference were still quite dark which indicates that there wasn't much difference at all. It's also easy to see that the biggest differences occured in places like the color bars. These types of areas aren't likely to occur in normal video footage. The areas that resemble normal video footage such as the leaves, the cookie, and the girl's face all showed the least amount of differences. This is easy to see in the "diff" pictures because they are black in those areas.
Chienworks wrote on 12/7/2003, 12:46 PM
Just for kicks, i repeated the experiment with the Microsoft DV codec. The files are in the same location and all start with "ms-". The difference is astounding! I would say that even the first generation render with the MS codec is worse than the 99th generation with the SONY codec. By the 3rd generation there was a noticeable mozaic effect, and by the 5th there were color/brightness changes as well as odd color streaks. By the 10th generation the leaves were almost unrecognizable and the colored text was mush.
mark2929 wrote on 12/7/2003, 1:28 PM
Kelly Thank you for doing this work I have been working with uncompressed Film with horrendous file sizes to try and maintain quality Magic Bullet recomends no generations is the best way and at the most only one. Would this test be valid on DV film blown up to cinema size
Chienworks wrote on 12/7/2003, 1:47 PM
Mark, well, if you could stay within the SONY DV codec you would probably be fine for multiple generations. However, the SONY DV codec isn't available to other software so that nips that idea in the bud. Uncompressed is probably still the best way to go.

I haven't heard much yet on high-resolution DV. It seems like all the hi-def stuff is going MPEG for greater compression.
mark2929 wrote on 12/7/2003, 2:00 PM
Doh silly me I got so carried away with looking at how good the sony codec was and the amazing tests you have done. I forgot I would be using other programs.

Ah well Ill have to save up for more hard drives

Thanks Kelly.
mark2929 wrote on 12/7/2003, 2:11 PM
Kelly wait all is not los.t A lot a times you can frameserve, and Im sure you could get the sony codec to work in AE. Perhaps if you had limited FX In other Programs I could just render that uncompressed. Anyway the fact that the sony codec is so good this could still save a lotta Uncompressed footage taking over my hard drives.
johnmeyer wrote on 12/7/2003, 4:28 PM
Kelly,

This is an amazing amount of work, and very useful.

One question. In looking at the AVI file, I notice that there is no motion. Shouldn't some portion of the test involve moving the object in order to see what sort of motion artifacts are created?
Jessariah67 wrote on 12/7/2003, 5:20 PM
I'm presuming that if "ignore third party codecs" is checked and "use microsoft DV codec" is unchecked in preferences, then we are using the Sony codec?
Chienworks wrote on 12/7/2003, 6:44 PM
John, probably true, but that would take enormously longer to do. I think the still results are a pretty amazing demonstration on their own though. For that matter i wouldn't worry about motion artifacts too much. MPEG shows some motion artifacts at 3Mbps, and some of this is due to the fact that many of the frames only contain partial data that modifies the previous frame. By the time MPEG's bitrate reaches 7Mbps or so almost all artifacts are gone. DV is encoded at 25Mbps and uses all complete frames, so any motion artifacts would be correspondingly tinier, possibly undetectable.

Jesseriah, yes, you are correct. It does require shutting down and restarting Vegas to make the switch.
johnmeyer wrote on 12/7/2003, 11:27 PM
I wasn't trying to suggest you re-do anything. And you are right: the test pretty much "says it all" about the differences between the MS DV codec and the Sony DV codec.

I have the Mainconcept DV codec. I wonder how it would compare? I suppose I could repeat your test. I assume you devised some way to automate the process?
Chienworks wrote on 12/8/2003, 5:09 AM
John, the start image is there as a .png file. All i did was put that on track 2, then place the counter digits on track 1 with a cookie cutter effect. After rendering, i replaced track 2 with the newly rendered file, incremented the counter, and rendered again. It was a little time consuming, but not at all difficult. I've uploaded the .veg file to the same directory. It's named generator.veg and it's a Vegas 4.0d file.

If you'd like to repeat the process, you can gather all the resulting frames and render them to a single DV .avi file (since it's DV -> DV this last step should be lossless) and email it to me. I can generate all the .png stills from that file. My address for these sorts of files is: testbench at vegasusers dot com. Thanks!
Spot|DSE wrote on 12/8/2003, 6:57 AM
To test for codec degradation, a single frame, chroma/luma values are all that's needed, we posted this stuff on the COW about 2 years ago, and then again with the SOFO codec came out, we also used an uncompressed tif, even though it was using the Quicktime reader, and a tga file.
Scopes, or at the least, a color picker works just fine for this, accompanied by the eye.
However, we only did 50 generations, thinking no one would ever logically recompress media more than 50 times.
SOFO/Canopus codec came out clear winners.
Considering that the application must be shut down and reloaded for each time through the process, this is a tremendous amount of work on Kelly's part. It took up several days of our time when we did this for SOFO way back when...Of course, that was also on a dual 800 system, tops for the time.
Chienworks wrote on 12/8/2003, 8:50 AM
Spot, interesting. Maybe something is different now in Vegas 4 than what you used when you did the test two years ago. Vegas 4 doesn't require shutting down and restarting the application for each generation. I was able to run both sets of tests through the 100 generations in about half an hour each. So it wasn't that much of an effort on my part, although it was mind-bogglingly boring!
Spot|DSE wrote on 12/8/2003, 9:03 AM
Yes, Vegas 4 is very different than Vegas 2 and 3. Would have been nice to do this that fast.
Boring isn't the word. A script to do this would have been nice.
johnmeyer wrote on 12/8/2003, 10:03 AM
Well, I tried doing the test with the Mainconcept 2.4.4. DV codec. What I am about to report has nothing to do with the Mainconcept MPEG encoder built into Vegas, and therefore anyone who reads what follows should not panic and think there is some problem with anything in Vegas. The following concerns the DV codec which is available for purchase direct from Mainconcept.

I found that after just one or two generations, the video had become noticeably dim. After ten generations, the video had faded to the point of being unwatchable. After 99 generations, the video was gone completely.

Wow, I thought, must be something stupid I did. So, I then tried the exact same test using the built-in NTSC DV template (i.e., the Sony codec). After ten generations, the video looked great. Thus, I was able to repeat Kelly's test and get the same results.

Hmmm. Must be a bug in the newest codec, I thought. It's only been out a few months. So, I uninstalled 2.4.4 and went back to a two-year old version. Did the test again. Same results.

I then called MainConcept U.S. tech support. The guy there was unaware of any problem.

Now I was getting very puzzled. I've used this codec for years, in other applications, and it has always produced stunning results. It is noted specifically for having virtually no genreration loss by those that have tested it.

I did one more test. This time I did the test using VirtualDub. I did only ten generations, but after the tenth generation, the video looked virtually identical to the original.

Conclusion: There is some interaction between Vegas and the Mainconcept codec that causes the video to go very dim.

Kelly, I can email the file to you, if you want it. I have taken just one of the renders (the 10th generation), converted it to a PNG file, and emailed that.
Chienworks wrote on 12/8/2003, 10:23 AM
John, i just got it. Thanks!
Something is definately going wrong somewhere. That sample looks like the contrast has been reduced some 80% or so and then darkened about 5%. It also has a very noticeable grid pattern appearing.

I uploaded the image here: http://www.vegasusers.com/testbench/files/generations/mc-bad-test-10.png and took the liberty of watermarking it with "For demonstration purposes only. This is not a real test." so that no one will use it as negative propoganda.

I hope you find out the answer to what's going on.
johnmeyer wrote on 12/8/2003, 12:03 PM
In addition to contacting MainConcept, I also sent a bug report to Sony. I'll let you know what I find out. I'm pretty sure I did everything correctly (since I was able to duplicate your NTSC DV results). I double-checked the track headers and the opacity levels (I even have scripts that do this automatically, but I checked manually as well).
johnmeyer wrote on 12/8/2003, 12:22 PM
I did further testing. If I render two frames instead of one frame, the first frame degrades just as badly as before, but the second frame degrades pretty much the same as the Sony NTSC DV codec.

Whatever the bug is, it looks like some sort of boundary condition.
Chienworks wrote on 12/8/2003, 2:03 PM
One thing i had to be wary of when doing the Microsoft codec test was that Vegas insisted on importing the single frames as still images rather than as video events, which of course meant that they came up on the timeline as whatever length i had defined for stills in Preferences. The closest i could come was 0.033 seconds which was just a hair short of a full frame (0.033366674... seconds). If i left the event at this length, Vegas would mix (resample) the 0.033 seconds of the video with 0.000366647 seconds of black and ever so slightly darken the image. I had to manually stretch the event out to a full frame each time. I suppose i could have set it for 0.034 seconds instead, had it overlap the frame boundry, and then rendered a loop region of just one frame. Oh well, hindsight is always 20/20.
taliesin wrote on 12/8/2003, 5:48 PM
I've did lots of similar render tests in the past, so it is very noticable: There's something going wrong in your test.

Looking at the MS-Codec:

This one does compress the color range of RGB 0-255 down to 16-235 when encoding. But same way it does expand the color range back when decoding!

Looking at the Sony Pictures DV codec:

It does NOT compress the color range when encoding and does NOT expand the color range when decoding.

Looking at the latest MainConcept DV codec:

You have the choice whether the codec should compress/expand color range or not. It's up to you to change the preferences.

Now - if you take a graphic into Vegas it's not the DV codec which does the decoding of it. A DV-codec can not touch a graphic file for decoding. So if you use the MS-Codec to encode the graphic into DV it will compress the color range down to RGB 16-235.
Now be sure your Vegas settings does NOT use the Sony Pictures DV codec but the MS-DV codec and watch the rendered graphic. It will be just fine! It's fine because now the MS codec does expand the color range back to what it was.

Now change the Vegas setting, disable the MS codec to be back using the Sony Pictures codec. Now the rendered file looks bad because the Sony Pictures Codec does not expand the compressed color range back to 0-255. But this is not the way to go!

If you have ever encoded a file using the MS codec, you MUST use same codec to decode the file! You have a compressed color range on encoding, so you need to expand the color range when decoding!

Similar it is when always giving back a graphic for the next render generation instead of a DV-AVI. You then always drop the decoding process. But you need the decoding and expanding process to be on the right side.

If you do a render test with DV codecs your input and your output always must be DV and you always have to stay on using exactly the same codec both for encoding AND for decoding.

And as for the MainConcept DV codec: Just change the settings to keep the color range and/or ensure MainConceptDV codec is used for decoding (instead of SoFo Codec).

Marco
taliesin wrote on 12/8/2003, 6:19 PM
>> Conclusion: There is some interaction between Vegas and the Mainconcept codec that causes the video to go very dim.

You maybe used the SoFo-Codec for DE-coding but the MainConcept-Codec only for En-coding!?

You must be very sure the MainConcept will be used for decoding too!

If you don't - the MainConcept codec always does compress the color space when encoding but it never get's the chance the expand the color space on decoding.

So you must be sure MainConceptDV was the latest DV codec you've installed on your system and you must have "Ignore third party codecs" unchecked in the Vegas preferences to ensure to use the MainConceptDV codec for DE-coding too.

Marco
johnmeyer wrote on 12/8/2003, 11:20 PM
Marco,

Thanks for the cogent explanation. Tomorrow morning, I'll check my "ignore third party codec" settings (doing a long print to tape now that I don't want to interrupt, and then to bed).

Kelly,

Vegas insisted on importing the single frames as still images. I ran into this as well, and set the duration to slightly more than one frame. I thought maybe this was the problem, and so I tried rendering five frames of video (it doesn't take any more time). As I indicated earlier, I still got the problem with the first frame. I think Marco's explanation describes well what is going on. If I do another test, I think I'll create a five frame AVI file and then use that as my generation 0.