I just got this same error also: The file is an unsupported format. I've been working on a documentary for 8 months, having started in Vegas 11, and upgrading to 17 mid-way through. I have over a dozen filename.veg files in my backup folder. The ones from earlier open up in 17.0 (they were likely the ones created in Vegas 11). But recent versions (and their corresponding veg.bak files) will not open in 17, altho they were created in 17. There are no updates for 17 currently (i'm on v17.0 Build 452) and I haven't updated much on my PC, except for Windows 10 mandatory updates. I'm wondering if something occurred there........I am currently screwed since weeks worth of work is inaccessible. Ugh.
I found my solution: I had recently changed a setting under INTERNAL PREFERENCES: I'd changed Enable So4 Compound Reader for AVC/M2TS from TRUE to FALSE, upon somebody's rec. to help with crashing issues when rendering. I changed this back to the default TRUE and my most recent .veg opened up. Interesting. jmm