Render stress tests

A. Grandt wrote on 4/22/2011, 2:52 AM
I've created and used a short test to try and stress a renderer into creating artefacts or finding encoding errors, such as the one that is plaguing the YouTube 1080p uploads.

The current edition:
[url=

Vegas source and files: [url=http://www.grandt.com/Vegas/RenderStressTest.zip]
Size: 321 KB.
The link contains .veg files for both VP9 and VP10.
The Belle-nuit test chart used in the second half was downloaded from [url=http://belle-nuit.com/testchart.html]

The idea is to have a 20-40 second render with various "panels" of anywhere from 5 to 10 seconds length. These should not tax the compositor in Vegas too much, as it is the Video codecs that is the target of this test.

I'd love to hear about ideas, additions (or removals of superfluous elements) and potential problems. At the moment the only movement is the time code, it would probably be an idea to have panels that stresses motion and fading.

I especially need help to make sure that the output is 100 % correct with regards to colours. Currently there are two colour correctors, one (sRGB->cRGB) on the test charts track and one (cRGB->sRGB) on on the Video bus.

The Audio click track could probably use some attention as well, in case someone with an audio background could come up with a test for that while we are at it :)
Testing for instance levels and positioning (stereo and/or 5.1).
-A.Grandt

Comments

farss wrote on 4/22/2011, 5:33 AM
Certainly a great start however there's nothing in motion to check de-interlacing and compression. It could also include the ISO 12233 chart so people could get a quanitative measure of resolution.

Bob.
A. Grandt wrote on 4/22/2011, 7:17 AM
As I said, I needed ideas for testing motion :)

As for the ISO 12233 chart, it is not exactly free of charge, though there is one that replicates it through SVG. [url=http://www.graphics.cornell.edu/~westin/misc/res-chart.html]
I have rendered this one to PNG at a sufficinetly high resolution to crop it to 2xHD: [url=http://www.grandt.com/Vegas/ISO_12233-reschart_crop.png]
The original render is here: [url=http://www.grandt.com/Vegas/ISO_12233-reschart_full.png]

Are the cropped one the one you would like included? The full makes little sense as the "targeting" area outside the test panel is meant for camera use.
farss wrote on 4/22/2011, 2:44 PM
"Are the cropped one the one you would like included? The full makes little sense as the "targeting" area outside the test panel is meant for camera use"

Just one H and one V trumpet with markers should be all that's needed. They're probably going to be pretty good for measuring resolution and seeing any aliasing. The grids don't quite do it for me.

Many years ago I created a veg file that was based on part of an old BBC/ABC test pattern that had an audio sync area. The original, from a hardware TPG, is just a moving vertical bar with two fixed arrows such that when the bar is aligned with the markers there's a pip. I enhanced that to add markers that showed how many frames off you were from sync. Something like that might be handy.

Bob.
A. Grandt wrote on 4/22/2011, 5:06 PM
I'll have to look at the moving sync indicator. I have seen these on the test screens on TV at times.

For the "Trumpets" I've cut and pasted the bits around, is this simething like what you are after? : [url=http://www.grandt.com/Vegas/Trumpets.png]
farss wrote on 4/22/2011, 5:21 PM
"For the "Trumpets" I've cut and pasted the bits around, is this simething like what you are after?"

Yes, perfect.

Bob.
A. Grandt wrote on 7/8/2011, 1:16 PM
VERY late answer here, I just slammed a few motion elements and the trumpets on a timeline. It's slow, but should at least gauge the renderer performance with teh trumpets, which are much larger then the 1080p HD and thus needs to be scaled down along with the render. At least that I what I hope they'll do :)

(made with 10e)
[url=http://www.grandt.com/Vegas/RenderStressTest Motion Elements 1.zip]

The zip contains the .veg and the elements.
A 102MB 1080p24 render using the default Sony AVC $ 14Mbps:
[url=http://www.grandt.com/Vegas/RenderStressMotion_element tests 1.mp4]

If this is not what you wanted, let me know :)

What are we seeing in the video?
1: 10 seconds of resolution "trumpets", no motion.
2: 30 seconds of resolution "trumpets", rotating 90 degrees counter clockwise. Linear keyframe motion.
3: 10 seconds of resolution "trumpets", no motion, rotated the 90 degrees.
: In the .VEG I have added a Zoom to 50% between 5 and 10 seconds of this segment. It might highlight artefacts when the space between previously visible areas get close to pixel resolution. might being the operative word :)
4: 20 seconds of a 4deg line burst circle. Rotating Counter Clockwise 1deg/sec
4: 20 seconds of a 4deg line burst circle. Rotating Counter Clockwise 1deg/sec, but with another similar circle op top, with no motion.

As I said, it's just a little test where I took the elements I had and slapped them on the timeline.
The Trumpets at a high resolution (1600x1600), the LineNova is not, it's only 780x708 and thus scaled up. If there is an interest for it, I'll make another at a higher resolution.
All of these elements are bitmaps, I have a feeling the best test result would be if I had had them as vector graphics, though resizing bitmaps is something an editor should be good at, it
Steve Mann wrote on 7/8/2011, 6:55 PM
What is the purpose here?
A. Grandt wrote on 7/8/2011, 11:39 PM
The propose in the first post was to see if we could create a fairly decent render test, helping us make better videos by being able to detect errors in rendering, and re-rendering, such as on YouTube. Including getting the correct colors.

Bob asked for motion to help detecting artefacts and "trumpets" to help determine changes in resolution I guess. The last test merely included a bit of both, It was my plan to include these elements in the "Stress Test" once we've determined what is needed, and that it works for what it's intended.

In the end it might not be a bad idea to have a short document describing the elements getting into the tests, what to expect, and what to look for.

Originally I started the tests as i was suspicious about YouTube's claims regarding their resolution in 1080p mode. I was proven right, and the right test showed this clearly, sadly it took YouTube over a year to fix it.
musicvid10 wrote on 7/9/2011, 9:50 AM
When you arrive at a final version with motion, would you upload a project-with-media file somewhere?

I'd like to adapt it to 720p to use in my codec shootout elsewhere on this forum. So far I'm just using a static Belle-Nuit 720 chart.

Are the primaries in the first composite 601 or 709? Are they identical to Belle-Nuit?
A. Grandt wrote on 7/9/2011, 10:08 AM
The primaries in the first panel are just the default PAL test chart from the Vegas generators.
The second chart is the 1080p version of the Belle-Nuit.

The Zip in one of my previous posts does contain the elements I have now. Especially the Trumpets are not going to change. How we end up showing them in the test project is a different matter :)