Try this codec (make sure you read up on it). I have not used it but while doing some reading up on this in the past I saved this one for its ability to do transparency (I believe they call them sprites).
you can apply a mask FX to the video & export a compressed video file from that (or a png image sequence). Then use that as the alpha channel for any future work.
I've started using the Lagarith Lossless Codec to shuttle video between AE CS3 and VP8. It supports alpha and saves disc space. Can be a bit tricky to install on Vista 64, but otherwise works great.