Wax 2.0 (www.debugmode.com) is a freeware Vegas 3D composting/FX plug in that supports DirectX and Virtual Dub video filters. You can also frameserve using Satish's Frameserver plug-in (same website) to any application that will take an .avi input.
I think Satish was working on an adapter for Photoshop and maybe After Effects plug-ins, but I don't know where he is on this.
> I know how it works with audio (VSt/directx), but doe sthis work for video?
Yes, as long as they are DirectX video filters. Pixelan SpiceMaster filters are one example. When you install them, they show up in your Video FX folder with all your other Vegas build-in filters.