Assuming one has captured 10-bit DigBeta via SDI and has an appropriate 10-bit codec installed, does Vegas process the video effects, titles and transitions as 10-bit or is it limited to strictly 8-bit video?
Not by Sony AFAIK. It was implied by Sony not saying "oh, it follows the depth of the highest common denominator in your timeline or media bin assets".
The clue is to look for contouring and be certain that your SDI is giving you YUV-8 and your capture is storing this as RGB-10. If it is YUV-10, you need to store that in YUV-10 or RGB-12.
The best critique is to visually (monitor or S:N test) analyse how Vegas manages with our sources.
If you keep your veg file, you can probably recapture in 10bit when you get Vegas6 anyway.
Perhaps Sony will mention this in the Vegas5 manual. To save us guessing and looking for contouring/registration faults?
If you use the full dynamic range of either type of video data presentation, then you'll have some considerations to make. However just as with audio, the chances are that you as the source creator give yourself some headroom. This is both appropriate and saves you worrying too much.
There is a formula to take RGB and pop it into YUV space. Also a consideration if you are forced to convert the colour subsampling, e.g. from NTSC DV 4:1:1 to MPEG/PALDV 4:2:0 for DVD-Video.
It just isn't a perfect translation as there are figures below the decimal point which can lead to quantisation noise in the transfer. ie Rounding errors. This can appear as contouring on gradients.
This technology results in an imperfect transfer but we are in an imperfect world. It is just as likely that the look is better after the effect than the calibration of the source. The fundamental information is very very similar.
The extra latitude of going from 8bit into 10bit for eiher type does make the transfer more accurate.
Of course, if your footage is black and white, there is no error to be had. It is a pity, IMHO, that the UV or RGB vectors can't be used to increase the dynamic range of black and white. Perhaps I am living in the past!
I hope this was what you meant by a couple of paragraphs. I've kept the math out as I am not in the best position to paste in the calculations here. I'd suggest a google if you want to study this further.