They don't give it out. Dreams of custom renderers were here for decades.

You can do custom encoder as vfw codec.
You can do extension with private rendering. Rendering is more or less controllable by Vegas extensions, you can render image of any video frame and use it to feed your private encoder.
You can write your own media generators that will import your codec into Vegas. It's doable through OFX or DX plug-ins.