This issue of "non-smoothness", the best way I can describe it, happens often. It's not Youtube doing it, it's the rendered version. Sometimes it's also a matter of the image being more pixelated, which seems to occur on dynamic parts rather than static. I'm using VegasPro15. It doesn't matter if I have nothing else running on my PC. It just happens in practically every video I render.
My PC won't allow the latest version to work, but if there's not a way to have a better, smoother result with my current version, maybe a later version has resolved this issue? You can see an example in the following video link, (go to 12:48) before the "non-smoothness" occurs.
I've been using Vegas for many years and have my version tweaked as best I know. It runs well during editing. It's only the final rendered product that lets me down in spots. My specs are listed on my profile FYI. Thank you for reading and any advice you may provide.