Testing codecs with non-static footage

LarsHD wrote on 6/9/2009, 3:29 AM

Hi, I'm about to put together a little test sequence that I can use to test various aspects of codecs.

Bob, what moving "non-static" picture material do you see as essential to include in such a test?

David N. what are your thought here, what picture material, still and moving do you think is important to test when comparing different codecs (like comparing cineform to mxf etc)?

a) a noisy image
b) an underexposed image
c) an underexposed image with lots of electrionic noise
d) fast panning motion
e) etc etc

What aspects do you see as essential when judging the quality of a codec?

Best & thanks in advance
Lars

Comments

farss wrote on 6/9/2009, 5:09 AM

"Bob, what moving "non-static" picture material do you see as essential to include in such a test?"

As much motion as possible. That's provided by having the greatest possible variation between frames. The trap is that DCT compression may be applied before temporal compression so each frame has to first stress the DCT compression. If you do some research (which sill save me a lot of typing and mistakes) you'll get some idea on how say mpeg-2 compression works and therefore how to test it.
Clearly noise is a very good test, it's very stressful on the DCT compression and the temporal compression.

Having said all that the results are highly subjective, there's no metrics for measuring the results as our perception of the moving image is very complex and not terribly well understood.

One thing to be understood is that some of the more recently developed encoding systems are designed to get as much performance as possible in the least bandwith. They may sacrifice some aspects of image quality such as resolution to avoid other artifacts that are visually more noticeable. This makes it even more difficult to evaluate. When you look at a single frame it might look really bad but when you playback the frames as intended they doesn't look so bad at all.

Bob.

LarsHD wrote on 6/9/2009, 5:49 AM

Thanks Bob

Lars

busterkeaton wrote on 6/9/2009, 7:28 AM

I would think of footage that has caused problems for you before.

A background of blades of grass or leaves on a tree in bright sunlight on a breezy day is something I have seen pixelate before, because there is such variation in light among the individual blades of grass/leaves. Then if they are moving in the breeze, the codec is stressed even more.

LarsHD wrote on 6/9/2009, 8:16 AM

Thanks for good ideas! :)

So fine patterns, resoution charts moving, fine crosses lines etc. Pin stripe suits, zooming into fine patterns of various stuff, texture like / textile like stuff, still and moving...

I've put together some 15 different scenarios here into one project so that I will something to test with here.... I've already noticed many interesting things now... Interesting comparisons ahead I hope!

Lars

reberclark wrote on 6/9/2009, 8:55 AM

I have tested panning and motion problems with:

Chain link fences.
Rock walls.
Bushes, closely planted trees, tall grass (still and with wind)
A moving train (goes right behind my house!)

I would say anything with a repetitive pattern, like your examples, would work.

Happy testing!

LarsHD wrote on 6/9/2009, 9:25 AM

Thanks all. I've made a project approx 5 min with lots of stuff now. Static. Moving, Motion, Zooming. Noise. Resolution. Moving, smooth moving things, rotating things, rotating res stars etc etc etc etc. Test stuff for levels, luma levels, chroma etc etc This is all very exciting. Codec manufacturers will offer me free trips to Hawaii (if I stop this).

ALready I see *very* big differences and its repeatable and easy to communicate now.

Lars

busterkeaton wrote on 6/9/2009, 9:28 AM

Cool.

Let us know your results.