I just came across this PDF document that describes all the settings in the standalone MPEG Mainconcept encoder. I thought others might find this useful in order to understand all those settings you see when you click on "Custom." Of course, you may go blind if you try to change any of these settings, so proceed at your own risk!

MainConcept MPEG Encoder Advanced Interface