The only reports I've seen of Vegas with GTX 10X0 cards have both been negative. See the spreadsheet linked to in this post. Also note that Vegas generally works much better with AMD's OpenCL than NVIDIA's. In other words, avoid that one.
The jury is still out on RX 5X0 cards. We know that the drivers available for it (17.XX.X) will almost certainly crash Vegas when the Defocus and Starburst FX are used with GPU acceleration. Also see this thread about RX 580 issues. You could try the RX 560 if Defocus and Starburst are not important to you. There's a good chance it will be fine otherwise. Or you could look for a more proven RX 4X0 card. The RX 480 is in short supply and currently very expensive but the lower-specced cards may be more available.
I wish Vegas would get with it on the Nvidia support, the radeon cards are so bad (mainly the drivers)... My system stability has improved 10 fold since I started using Nvidia cards again.
I had a Radeon HD 6870. I do video projection mapping which requires me to output to multiple projectors at once. At two different jobs, as soon as I'd fire up the Radeon's displays to get ready to start mapping, it crashed on me so badly that I couldn't even get into windows, nearly ruining both gigs. I was 30 minutes late starting one job because of it (and I had to leave one projector off to get it working), as the band played on stage with no visual effects at all. The other I had to shut down early before the final band had played. After that, I moved on to a GTX 970, which while Vegas doesn't support it at all, is a far more powerful card that has been rock solid with stability, even when outputting to even more projectors at higher resolutions. I'd also had similar issues with other older radeons in the past. The drivers for AMD also were harder to deal with. I'd have more issues with overscan, my image would not fill the projector screens and the settings that were supposed to fix it wouldn't work, so I'd end up having to put the projectors farther back to fill the area I was mapping, and the GUI for the catalyst control panel would give me other issues from time to time as well. Never the case with NVidia.
Vegas uses OpenCL for timeline rendering of FX and compositing too. Nvidia's implementation of OpenCL is just not as good as AMD's. That means AMD cards are better for timeline performance and rendering but neither are supported like the older cards by MC AVC for rendering.
I gather there is no perfect solution to an upgrade. With my understanding it seems better render times can be obtained with certain AMD cards but poor video previews in Vegas. nVidia cards are a bit slower (?) but better previews in Vegas. True?
No. If at all it is the opposite. AMD cards are better for preview and will help even with rendering. Note that I said help because the only two codecs that supported full GPU accelerated rendering where Sony AVC and MC AVC. Both however are old encoders and written for old cards, GTX580 and HD6970 where the last cards that where supported. MC AVC has an option to chose between CUDA and OpenCL, Sony AVC is OpenCL only. Again this only works for the old cards as mentioned earlier.
There must be a thousand or more posts about this topic.
So yes, there is a "good" path forward. Buy the fastest 6-8core CPU you can afford, an AMD card like the RX480 or a Fury X, 32GB RAM and in my opinion skip mechanical drivers for anything but long term storage.