AVC (H.264) will give you the smallest file size for any given resolution and given visual quality (bitrate). Bitrate determines the size of your file regardless of resolution. The lower the bitrate the lower the visual quality.
It's been well over a year since I've rendered out to anything other than XDcam .MP4. The quality is outstanding, the files are relatively compact, and they can be shrunk in Handbrake perfectly for Youtube or Vimeo delivery. As much as I've tried to see a quality difference between XDcam .MP4 and Cineform or DNxHD, I simply can't. My eyes aren't that good.