For AVI files Vegas uses Video for Windows. It does not use it's own internal codecs for AVI files. You will need to install an AVC Video for Windows codec to read these files.
AVC in AVI files is a little unusual, but it comes up at times. You could re-mux the AVI file, using Avidemux or similar, into an MP4 and that should work and then you can avoid the codec hell that can be AVI.
Vegas does not support MKV. It does support MP4 but only with more common contents. AVC should not be a problem. What is the audio format in the source video file.
Handbrake output should not be a problem to import in V12/13. What settings in HB did you use? Video and Audio. I could try those and see if it loads for me.
I have just used Handbrake with the video codec set to H.264 (x264) with all other settings left as default and the file now loads into Vegas 12/13.
It was only a few weeks ago I was almost tearing my hair out as every file I tried either refused to load after conversion or loaded but with audio only and a green video.
The only thing I can think of that's changed (apart from installing the trial of 13) was an update to the AMD video drivers this morning. But I haven't even performed the required reboot yet.
What NormanPCN said. You can use the command-line version of ffmpeg to copy the video stream and convert the audio to AAC with a line something like this:
I'm just learning how to use ffmpeg myself so don't ask! It just happens that by coincidence I have the same problem. The converted file will have the original h264 stream and an AAC audio stream, and Vegas 12 Pro recognises this as "Sony AVC".