Simple.
H.264 is commercially licensed by MPEG-LA.
x264 is open-source, GNU/GPL standard license.
Handbrake, of course, uses the latter, but the names are more-or-less interchangeable.
They both do the same thing.
If you will be encoding HD below 12Mbps, I suggest x264 in Handbrake as giving you the best bang for your bits. They are both great at higher bitrates, but Mainconcept H.264 falls apart at lower average bitrates.