Speech to Text via Whisper openAI

bitman wrote on 11/11/2022, 9:40 AM

Some good news, I have been diving into an alternative to "Vegas Pro Speech to Text". Although it is a very fine feature and easy to use, it has some drawbacks: first of all, it is a Vegas 365 - only feature (I hope this may change). Lots of people will not be able to use it - those without a subscription. Like all AI based stuff it is dependent on the model and results may vary. It also lacks a way to tune for quality and language. I suspect it also being more favorable to English.

So here is an alternative: Whisper openAI.

I have created a simple Vegas script to call whisper and convert speech to text. Just place the cursor over an event on the timeline and the script will create result files with text. In a future version I can extend this to create subtitles from these result files on the timeline, feel free to add this or add more of the whisper capabilities like quality, language and translation options. Refer to the document on Whisper at the bottom of this post.

Latest Update 18/12/2023:

Whisper Speech To Text v7":

add backward compatibility for (old "Sony" Vegas versions UI plugin naming) scripting in Vegas 14,15,16 (Note: only tested in Vegas 21 - not tested in 14, 15 or 16)

Here is the link to the latest Vegas script called "Whisper Speech To Text v7":

https://www.dropbox.com/scl/fi/bpj52eo7hjoqda78upi9u/Whisper-Speech-To-Text-v7.cs?rlkey=l0elg1wjl3qi4051uuuscwdws&dl=0

older scripts:

Whisper Speech To Text v6":

sets timeline timecode format to "Time" to prevent subtitle discrepancies on the timeline and reset back to the original user's preference after the subtitle insert

Here is the link to the latest Vegas script called "Whisper Speech To Text v6":

https://www.dropbox.com/scl/fi/5t7pdc16rd3ab65ey2a3r/Whisper-Speech-To-Text-v6.cs?rlkey=y3ueerkpurf6o03amobf3xzkr&dl=0

"Whisper STT RAW v2" (variant):

same script as v6 with the exception that it keeps the original sentence layout "as is" of the .srt file without automatically adding a newline (like a word wrap) after 9 words.

Here is the link to the latest Vegas script called "Whisper STT RAW v2":

https://www.dropbox.com/scl/fi/nagvbfttmeus4axu88aon/Whisper-STT-RAW-v2.cs?rlkey=ok9lx6bfs9sqso62gqgp9hqbf&dl=0

====================================================================================

The only caveat is that it requires quite a bit of effort to get whisper installed, it depends on Python, GIT, FFmpeg, etc. and setting of environment variables. So, you need to install a bunch of supporting stuff before you can use whisper. But it is doable. For this purpose, I have put together a document on how to use and install whisper (and its dependent programs), it has all the links to get you up and running.

Here is the link to the document on Whisper openAI:

https://www.dropbox.com/s/dh62ripb58xth86/AI%20whisper.docx?dl=0

==================================================================================

update history

Update 18/12/2023: "Whisper Speech To Text v7"

add backward compatibility for (old "Sony" Vegas versions UI plugin naming) scripting in Vegas 14,15,16 (Note: only tested in Vegas 21 - not tested in 14, 15 or 16)

Update 14/12/2023: "Whisper Speech To Text v6" & "Whisper STT RAW v2":

sets timeline timecode format to "Time" to prevent subtitle discrepancies on the timeline and reset back to the original user's preference after the subtitle insert

Update 13/12/2023: "Whisper Speech To Text v5":

Compatibility update: solves subtitle insert fail issue some users have because of different results from whisper:
for those that whisper saves filename + media type extension + .srt
for those that whisper saves filename + .srt (without media type extension .wav .mp4 etc...)

On request here is a variant of the "Whisper Speech To Text v5" script called "Whisper STT RAW v1":

same script as v5 with the exception that it keeps the original sentence layout "as is" of the .srt file without automatically adding a newline (like a word wrap) after 9 words.

Update 11/12/2023: "Whisper Speech To Text v4":

Small update (but also a big improvement and bug fix for some users to support speech to text when the drive location of the audio media is not located on the same drive as the Vegas project.

Update 10/12/2023: "Whisper Speech To Text v3":

Very small update to support audio filenames with spaces in their names

Update 29/11/2022: Whisper Speech To Text v2":

Major update: I made a new improved script with UI to select different transcode model options, a translate option, and a UI option to import the subtitles from the generated files to a new track

11 November 2022: Original script "Whisper Speech To Text":

Have fun!

Back to post

Comments

bitman wrote on 5/18/2024, 2:43 AM

@Kilo if you take the latest script (Whisper Speech To Text v7) and open it with notepad++ then go to line 669, there is a statement to check for spaces between words to determine the number of words before a line break. It is currently at value 9, meaning 9 words, which seems optimal for English in normal horizontal aspect. For vertical videos you can perhaps lower this value to 2 or 3... Feel free to experiment!

Kilo wrote on 5/18/2024, 3:59 PM

Thank you so much bitman, I know you don't have to help but just know this is a tremendous time saver for me and my business. Unfortunately I can not get it to work. when I change line 669 of the code it seems to make the words go below each other

kind of

like this

but i was hoping that I would have 2-3 words per text media generator on Vegas pro. I'm not sure if this is possible but if you have any suggestions I would be forever grateful! Thanks again!

Kilo wrote on 5/20/2024, 3:00 PM

Ive spent days trying to figure this out and have made almost no progress. I need to find a way to shorten the amount of words to 2-3 per vegas pro text section! If anyone can help in any way or even just suggest a possible solution i would do ANYTHING! ILL FOR REAL PAY SOMEONE! pls help <3

bitman wrote on 5/21/2024, 1:07 PM

@Kilo To limit the number of words without newline breaks, then the whisper generated .SRT internal contend needs to be split (including timestamps) before import. This can be achieved by extra new code in the script, or separately by an external application.

This could be a nice project, but I do not have the time to do it now. It is more like a winter project!

If you take the latest script (Whisper Speech To Text v7) and open it with notepad++ then go to line 295, you can change the font and lower the font size so you can cram more words on your vertical videos... Not really a solution, but I suppose all small bits help!

Kilo wrote on 5/22/2024, 1:42 AM

After awhile of messing around i found that if you add

"sw.WriteLine("whisper " + "\"" + myFile + "\"" + " --word_timestamps True --max_line_width 16 --max_line_count 1 " + modelOption);"

On line 162 instead of what is currently on 162 in V7 it will create srt. files that has sections of no more than 16 characters (or whatever you change that number to)

Thank you to bitman again, if you ever do continue this project i hope this helps u slightly. I may continue to improve this script to add a few other easy customizations in the future and i will post here with the results if i have any positive ones

bitman wrote on 5/22/2024, 1:02 PM

@Kilo Thanks for the tip, I suppose vertical videos as a final delivery are getting more popular for smartphone end use!

As a side note, it is a PIA to camouflage and integrate vertical video into 'normal' horizontal FHD aspect in a more or less visually satisfying fashion.

bitman wrote on 6/28/2024, 8:11 AM

Some new Vegas 21 builds (e.g build 315) may have issues with my script v7, I have a new version available v8 which checks on the Vegas main version only rather than main version + build version, otherwise it gets confused with Sony :)

here is the link to v8

https://www.dropbox.com/scl/fi/voojqanq3cz0a7le31e2r/Whisper-Speech-To-Text-v8.cs?rlkey=xeq7v2zlqk8ir0aq154ewv0sw&dl=0