VMS17 was released some 3 years ago well before the release of Windows 11 and thus does not natively support Windows 11. Try uninstalling/re-installing or setting back to factory settings via: https://www.vegascreativesoftware.info/us/forum/faq-how-can-i-reset-vegas-pro-to-default-settings--104646/

VMS(P)17 was the last ever version of Vegas Movie Studio so there is no upgrade to a later version of VMS. The closest other product is Vegas Pro which is the big brother of VMS. The next version of Vegas Pro, 20, is iikely to be released sometime during August. Vegas Pro always offers a trial version so you can see if it is good for you.