Why does Handbrake look so much better?

Laurence wrote on 12/9/2010, 2:08 PM

I've been messing around with different h.264 encoding methods: Vegas, Handbrake, Badaboom, Media Espresso, etc.

What I'm finding is that Handbrake gives me the best looking results at any given bitrate. At low bitrates the difference is just stunning. At higher bitrates, the difference is less, but still easy to see even with casual viewing. This is with progressive shot and edited projects, so it isn't because of Handbrake's excellent decombing option.

My question is, why does Handbrake look so darned good? and is there anything else that is as good or better? and are their any GPU accerated options that look as good as Handbrake?

And one more question: in order to transcode with Handbrake, I need to use uncompressed or mpeg2 (like m2tor .mxf). It is kind of frustrating to have to do a large uncompressed render then transcode with Handbrake in order to get the highest quality on my Vimeo uploads. Does anyone have a better way to do this?

Comments

Tom Pauncz wrote on 12/9/2010, 2:15 PM

Perhaps frameserving out of of Vegas would save uncompressed encoding step?

Just thinking ...

Tom

Laurence wrote on 12/9/2010, 2:28 PM

I just found this:

http://forum.handbrake.fr/viewtopic.php?f=6&t=11258

It seems that Handbrake performs best if it can do an analysis pass first, which precludes frameserving.

farss wrote on 12/9/2010, 2:43 PM

I gave up on Handbrake, I can get as good if not better results out of Vegas without all the extra work. You need to do a careful de-interlace in Vegas though. Adding in a denoise FX can help too.

Bob.

Jøran Toresen wrote on 12/9/2010, 2:53 PM

Bob, could you please describe in more detail what you mean by ” a careful de-interlace in Vegas”?

Jøran

PerroneFord wrote on 12/9/2010, 2:58 PM

This is why I use Squeeze. Great results, simple workflow. Sometimes you really do get what you pay for. Procoder used to be good too, but it's now wrapped into the Canopus product apparently.

willqen wrote on 12/9/2010, 3:02 PM

Yes, Bob. Please reply and let us know how you get the "as good or better than Handbrake" h.264 encode quality out of Vegas. Sure would be a huge help to me and others too, I presume.

I appreciate your time and knowledge,

Will

farss wrote on 12/9/2010, 3:31 PM

I use Mike Crash's free Smart De-interlacer.
As others (John Cline being one) have said this is a bit of a bear to setup correctly and I do not believe there is a one shoe fits all answer. The other aspect of it is if you get it wrong you leave artifacts only a frame or two long which get much worse with any high compression codec e.g. H.264 / AVC.
To get all the parameters in the de-interlacing set as well as they can be I always look for a couple of frames with isolated fast motion e.g. a hand waving. As you change settings you will not see the outcome unless you slide the playhead back and forth a couple of frames. This is a temporal FX and it needs to have several frames of vision sent to it to work.

I feel fairly confident in saying the reason Handbrake can produce superior results is not because the encoder is better but because of the preprocessing it does before the encoder. Vegas does not hold our hands. To get the best out of Vegas you have to work at it.

As a starting point here is a link to a post by Glenn Chan on using this FX with Vegas: http://www.dvinfo.net/forum/what-happens-vegas/52097-mike-crash-de-interlace-filter-notes.html

Two settings I have found may make a difference are covered by Glenn thus:

Blend instead of interpolate in motion areas: I would check this. Blend fields gives higher resolution.

Look carefully at what the FX is doing and try changing these parameters. Artfifact free is better than maximum resolution.

Bob.

musicvid10 wrote on 12/9/2010, 4:16 PM

x264 outperforms MainConcept in quality and rendering speed in every test I've seen (YMMV).
At high bitrates the quality differences can be negligible, but the speed differences are still there.
And for those of us using the older MainConcept (Vegas 8), there's quite a difference.

And at low bitrates, the quality difference can be night and day.
Two major advantages are the decomb option (mod yadif), and the Constant Quality Rate Factor, which is as good or better than 2-pass VBR. That method is not a good predictor of file size, however, because of the wide variations in source footage.

Alas, HB will not accept frameserving because of the analysis pass. I, like Laurence, did a lot of experimentation for the perfect intermediate, and hit a brick wall with many (DNXhD, Lagarith, Huffy) because of the 709 colorspace error that the HB folks don't seem to be able to fix. Handbrake won't touch Cineform. I've had very good luck however, using Sony MXF as an intermediate. For more on this, see:

http://forum.handbrake.fr/viewtopic.php?f=11&t=17364

Laurence wrote on 12/9/2010, 6:58 PM

Yeah, .mxf works well and you can set the bitrate insanely high and still have a major space savings over uncompressed. I really wish Cineform worked with Handbrake. Has anyone tried the Avid codec?

Sorensen Squeeze looks good but it is expensive and as far as I know, has similar input codec limitations to Handbrake. Squeeze is a great option for better SD DVD renders, but Handbrake does equally good with h.264 and I find myself doing less and less DVDs these days since Internet video delivery looks so much better.

My experiments tell me that Handbrake looks markedly better and more detailed than anything I can render out of Vegas with either of the h.264 codec options. That is with progressive material from beginning to end so it is not because of the deinterlacing options. As I said before, at low bitrates the difference is stunning, and at high bitrates, the difference is less but still quite easy to see.

NickHope wrote on 12/9/2010, 7:08 PM

Leaving hardware acceleration aside, does anyone know if there has been a quality improvement in either of the bundled H.264 codecs (Sony or MC) between 8.0 and 10.0?

Laurence wrote on 12/9/2010, 7:12 PM

No quality improvement that I can see. Still the same "almost good enough" look that I just can't bring myself to use.

Laurence wrote on 12/9/2010, 7:32 PM

I just tested .mov format with png lossless compression and Handbrake will accept that. That may be the best option for now.

edit: maybe not, the colorspace is messed up and the conversion from the png .mov is lacking in contrast. Lots of detail, but the colorspace looks wrong.

farss wrote on 12/9/2010, 8:24 PM

Can I ask specifically what are we talking about here?
I'm encoding to 720p using the Sony encoder at 6 or 10Mbps which I consider low but that's subjective.

Bob.

musicvid10 wrote on 12/9/2010, 8:33 PM

"Has anyone tried the Avid codec?"
Yes, as mentioned in the link, the DNxHD codec has the colorspace error as well when rendered in Handbrake.

Of course, if the final destination is Youtube or Vimeo, that may not be such a bad thing, since both services will expand it back out to 0-255. Might be worth one of my sometimes-revealing grayscale tests.

Hulk wrote on 12/9/2010, 8:51 PM

I use VP10 with good results but I do fiddle with the de-interlacing parameters a bit as Bob wrote.

Isn't it kind of silly that in this time of high definition video progressive (LCD) display devices we are still dealing with interlacing? And even though most cameras can output progressive video it's still wrapped in an interlaced stream?

I don't know, maybe it's me but I have yet to be able to "unwrap" the video from my Canon HF100 and see the original true progressive frames when editing in VP10.

Sorry for the OT post but if someone knows how to do this please inform me!

- Mark

Laurence wrote on 12/9/2010, 9:06 PM

>Can I ask specifically what are we talking about here?

What I see (looking at results from tests I just did today) is that a handbrake render at 3Mbps looks quite a bit better than a Sony render at 6Mbps. With a Vegas render, I have to get closer to 10Mbps before I start liking it. With Handbrake I'm already there at 3Mbps. That's 1280x720x29.97 progressive from 1440x1080x29.97 progressive HDV. Renders are from the current version of Vegas.

Steve Mann wrote on 12/9/2010, 9:38 PM

Would it be asking too much here for a step-by-step workflow for using Handbrake to encode h.264 files?

musicvid10 wrote on 12/9/2010, 10:23 PM

OK Laurence, here goes:
--All renders were 720 30p to keep deint. out of the equation.
--Rendered DNxHD in RGB space; confirmed that levels held.
--Took DNxHD into Handbrake, rendered to x264. Result was flat, 709 conversion error had occured!
--Uploaded flat MP4 to Vimeo, was converted back to RGB space.

Video is here: http://vimeo.com/17665395

Here's how it went:

Original DNxHD render from Vegas (RGB space)

Handbrake x264 output from DNxHD (colorspace error)

Vimeo processing result -- converted it back to 0-255 (also lifted the gamma some ;?()

musicvid10 wrote on 12/9/2010, 10:30 PM

@Steve Mann

Here is the original Flash tutorial I did for Randy Brown on using Handbrake. Only real difference is the frame dimensions (we were working with Randy's SD material):

http://www.mediafire.com/?l8i6leq5j0tnhhc

Post back if any ?

Laurence wrote on 12/10/2010, 8:01 AM

MusicVid, That is pretty neat. I think however that what Vimeo does is simply look at the the levels and sort of normalize them. For a while I was uploading a sort of half cRGB preset with blacks of 0,0,0 and whites of 235,235,235 until I realized that no matter what I uploaded, Vimeo was going to scale it accordingly. Now I just make an h.264 out of my regular sRGB colors. I am worried that if I screw up the color space and let Vimeo correct it that I will have problems with moire on shaded surfaces and skin tones.

I did some more experiments and find that Handbrake messes up the color space of uncompressed as well. I can compensate for this by putting an sRGB to cRGB color corrector before I do my uncompressed or Avid codec render and then the color space is pretty close on the Handbrake render. I think that just rendering to HDV or .mxf is really the best looking and most practical solution for now though.

Laurence wrote on 12/10/2010, 8:07 AM

Here is a discussion by FCP users going through exactly the same thing:

http://www.dvxuser.com/V6/showthread.php?222134-Which-codec-should-I-export-from-FCP-for-Handbrake-to-compress

musicvid10 wrote on 12/10/2010, 9:30 AM

"I think however that what Vimeo does is simply look at the the levels and sort of normalize them."

Nothing quite as elegant as that. Vimeo and Youtube simply apply a stock sRGB->cRGB to everything that is uploaded, whether it needs it or not. IOW, anything with RGB levels (or greater than 16-235) will get blown out. It's a shame that Vimeo doesn't trust its members enough to pass levels untouched, wsywig.
http://vimeo.com/14151131

musicvid10 wrote on 12/10/2010, 9:43 AM

I submitted a feature request to Handbrake some time ago to include a "Force RGB Levels" option, along the lines of this code, which works in AVISynth:

ConvertToYV12(matrix="PC.709")

Unfortunately, the suggestion came back marked "Invalid" (their developers do seem a bit cranky at times . . .).

musicvid10 wrote on 12/10/2010, 9:48 AM

It's looking like Handbrake is going to mess up YUV regardless, but handle MPEG-2 based encodes well. I'll dig into it a bit more over the holidays. Thanks for an interesting discussion.

EDIT: Laurence, try this, and see if you can confirm my results.
I hadn't thought of it, because it seemed counterintuitive, but maybe I was thinking backwards (wouldn't be the first time).

-- Start with a 720 30p 8-bit project in Vegas.
-- Render to DNxHD setting the colorspace to 709, and using a 720p 8-bit template.
-- Take the DNxHD into Handbrake, and render using x264, 720p.

Tell me if the levels hold firm, and I'm not seeing things!
Thanks.