Vegas does not support all high profile parameters inside Handbrake. The results will look terrible.

Here's what you should do:
8x8 DCT = Off

The most you will lose is ~2% compression from doing this, and then only on high-motion footage. Quality will not be affected.

Here, is my Normal-Dumb preset that should work with anything, assuming sane bitrates. Note that most of that high profile stuff is useless at our level:

ref=2:bframes=2:subq=6:8x8dct=0:trellis=0:weightp=0:b-pyramid=none:deblock=-1,0

If you really want some screaming encode times in Handbrake, make your own Superfast preset using the docs on their website.