I believe the RX 5700XT is already supported but it just doesn't provide a boost in Vegas performance as some older cards... It should appear as a choice in Vegas under Preferences/Video/GPU Acceleration & then in the "Render As" options, you should see Nvidia NENC as a render choice...
The RX 5700XT, with only 2560 Stream Processors, is the latest generation mid-range card designed for better gaming vs. significantly better compute / content creation power... IMO it's better than the similarly priced RTX 2060 in my laptop, which is better than an AMD RX 5xx series card...
The previous generation 2080ti, with 4,352 processors, Radeon 7 with 3084, or VEGA 64 with 4096 should still have better performance in Vegas... IMO, a last-generation, discounted VEGA 64 for $350 is the current best bang/buck with Vegas17, but it needs more power than the latest generation of GPUs. Plus, upcoming high-end cards from both Nvidia & AMD should provide much better performance and/or drive down the prices of the 2080ti & Radeon 7... Note - my liquid-cooled VEGA 64 has 13.7 TFLOPS of GPU performance vs. the 5700XT's 9.0 TFLOPS of performance...
IMO they will be discontinuing the VEGA 56 / 64 soon when new high-end cards replace the Radeon 7... Then the Radeon 7 will be discounted below its current $550 price... If the VEGA drops below $200, I'll get a couple more for my older Xeons & maybe upgrade my 9900K to a discounted Radeon 7 or something better...
The cost of HBM2 RAM might keep Radeon 7's price not far from where it is currently, but it has already dropped below what AMD said was what they could sell it for profitably, so who knows.