Speech to Text via Whisper openAI

bitman wrote on 11/11/2022, 9:40 AM

Some good news, I have been diving into an alternative to "Vegas Pro Speech to Text". Although it is a very fine feature and easy to use, it has some drawbacks: first of all, it is a Vegas 365 - only feature (I hope this may change). Lots of people will not be able to use it - those without a subscription. Like all AI based stuff it is dependent on the model and results may vary. It also lacks a way to tune for quality and language. I suspect it also being more favorable to English.

So here is an alternative: Whisper openAI.

I have created a simple Vegas script to call whisper and convert speech to text. Just place the cursor over an event on the timeline and the script will create result files with text. In a future version I can extend this to create subtitles from these result files on the timeline, feel free to add this or add more of the whisper capabilities like quality, language and translation options. Refer to the document on Whisper at the bottom of this post.

Latest Update 18/12/2023:

Whisper Speech To Text v7":

add backward compatibility for (old "Sony" Vegas versions UI plugin naming) scripting in Vegas 14,15,16 (Note: only tested in Vegas 21 - not tested in 14, 15 or 16)

Here is the link to the latest Vegas script called "Whisper Speech To Text v7":

https://www.dropbox.com/scl/fi/bpj52eo7hjoqda78upi9u/Whisper-Speech-To-Text-v7.cs?rlkey=l0elg1wjl3qi4051uuuscwdws&dl=0

older scripts:

Whisper Speech To Text v6":

sets timeline timecode format to "Time" to prevent subtitle discrepancies on the timeline and reset back to the original user's preference after the subtitle insert

Here is the link to the latest Vegas script called "Whisper Speech To Text v6":

https://www.dropbox.com/scl/fi/5t7pdc16rd3ab65ey2a3r/Whisper-Speech-To-Text-v6.cs?rlkey=y3ueerkpurf6o03amobf3xzkr&dl=0

"Whisper STT RAW v2" (variant):

same script as v6 with the exception that it keeps the original sentence layout "as is" of the .srt file without automatically adding a newline (like a word wrap) after 9 words.

Here is the link to the latest Vegas script called "Whisper STT RAW v2":

https://www.dropbox.com/scl/fi/nagvbfttmeus4axu88aon/Whisper-STT-RAW-v2.cs?rlkey=ok9lx6bfs9sqso62gqgp9hqbf&dl=0

====================================================================================

The only caveat is that it requires quite a bit of effort to get whisper installed, it depends on Python, GIT, FFmpeg, etc. and setting of environment variables. So, you need to install a bunch of supporting stuff before you can use whisper. But it is doable. For this purpose, I have put together a document on how to use and install whisper (and its dependent programs), it has all the links to get you up and running.

Here is the link to the document on Whisper openAI:

https://www.dropbox.com/s/dh62ripb58xth86/AI%20whisper.docx?dl=0

==================================================================================

update history

Update 18/12/2023: "Whisper Speech To Text v7"

add backward compatibility for (old "Sony" Vegas versions UI plugin naming) scripting in Vegas 14,15,16 (Note: only tested in Vegas 21 - not tested in 14, 15 or 16)

Update 14/12/2023: "Whisper Speech To Text v6" & "Whisper STT RAW v2":

sets timeline timecode format to "Time" to prevent subtitle discrepancies on the timeline and reset back to the original user's preference after the subtitle insert

Update 13/12/2023: "Whisper Speech To Text v5":

Compatibility update: solves subtitle insert fail issue some users have because of different results from whisper:
for those that whisper saves filename + media type extension + .srt
for those that whisper saves filename + .srt (without media type extension .wav .mp4 etc...)

On request here is a variant of the "Whisper Speech To Text v5" script called "Whisper STT RAW v1":

same script as v5 with the exception that it keeps the original sentence layout "as is" of the .srt file without automatically adding a newline (like a word wrap) after 9 words.

Update 11/12/2023: "Whisper Speech To Text v4":

Small update (but also a big improvement and bug fix for some users to support speech to text when the drive location of the audio media is not located on the same drive as the Vegas project.

Update 10/12/2023: "Whisper Speech To Text v3":

Very small update to support audio filenames with spaces in their names

Update 29/11/2022: Whisper Speech To Text v2":

Major update: I made a new improved script with UI to select different transcode model options, a translate option, and a UI option to import the subtitles from the generated files to a new track

11 November 2022: Original script "Whisper Speech To Text":

Have fun!

Back to post

Comments

bitman wrote on 12/14/2023, 4:39 AM

@Vegas_Pro_Brasil @jetdv Good find, I never realized that one of your issues issue was related to a specific user setting of the timeline timecode. Mine was set to yet another variant: "Time & Frames", this one however pretty much behaved like "Time" for the scripts. To avoid issues and to improve the script, I have added @jetdv code to set the timeline timecode to "Time" when adding subtitles, and restoring the user preference after the insert.

See start of this post for script version v6 (and the RAW variant v2), I have removed the links of the older scripts to clean up a bit.

Vegas_Pro_Brasil wrote on 12/14/2023, 8:19 AM

@bitman

I think now I owe you a case of beer and another case to @jetdv. Everything working as expected now. Thank you very much to you and Edward.

zzzzzz9125 wrote on 12/14/2023, 10:28 AM

hey, I've found two problems.

1. In Vegas Pro 16 and before, the script can't generate Titles & Text events properly for its GUID has been changed after 17. Just change {Svfx:com.vegascreativesoftware:titlesandtext} in your script to {Svfx:com.sonycreativesoftware:titlesandtext}, so that it can be used in 16 and before, without affecting the functionality in newer versions.

2. When I click the Balanced button, it can generate the .srt file (with other files) normally, but when I click Draft(fast), nothing is generated in the folder. I don't know what's going on.

bitman wrote on 12/14/2023, 11:56 AM

@zzzzzz9125 Thanks, useful information for those who are at Vegas pro 16 (or even older versions), feel free to change the script for yourselves, but I am not going to do it, for the simple reason that I cannot test it anymore on version 16; I upgrade every year and keep maximum 1 or 2 prior versions after the update; oldest version -1 usually get the axe when I upgrade!

jetdv wrote on 12/15/2023, 9:14 AM

@bitman, for my scripts that need to work in "14 and newer", I did this:

            if (myVegas.Version.Contains("14") || myVegas.Version.Contains("15") || myVegas.Version.Contains("16"))
            {
                genUID = "{Svfx:com.sonycreativesoftware:titlesandtext}"; //Sony Titles & Text
            }
            else
            {
                genUID = "{Svfx:com.vegascreativesoftware:titlesandtext}"; //Magix Titles & Text
            }

bitman wrote on 12/18/2023, 2:51 AM

@jetdv Thanks, I added the code in v7

Update 18/12/2023: "Whisper Speech To Text v7"

add backward compatibility for (old "Sony" Vegas versions UI plugin naming) scripting in Vegas 14,15,16 (Note: only tested in Vegas 21 - not tested in 14, 15 or 16)

@zzzzzz9125 version v7 for you to try, I could not test it on these older non Magix Vegas versions!

Vegas_Pro_Brasil wrote on 12/18/2023, 6:49 AM

@bitman @jetdv @zzzzzz9125

I tested it on Vegas Pro 14 that I have installed on my system and everything works correctly.

I have been studying this code of this script patiently and realized that it is very well designed.

Bitman took care to make it compatible with versions of Vegas where it is not possible to import srt files natively as text events and this is a very good thing, as it shows respect for users who have older versions

To my surprise, I managed to adapt this script to work with a Whisper variant called Whisper-Faster which is about 5 times faster than the standard Whisper and everything is working correctly.

I created a Whisper-Faster Windows installer and adapted the script created by Bitman and too other scripts and presets to work in Vegas. Below is a small demonstration of how it works. There are still some small improvements left but it is almost finished.

bvideo wrote on 12/19/2023, 3:18 PM

Fantastic!

pierre-k wrote on 12/20/2023, 4:36 AM

I know that if I want to export subtitles from Vegas to srt or sub, I have to use Vegasaur.

Has anything changed for the better over the years? How do you export these subtitles?

Vegas_Pro_Brasil wrote on 1/15/2024, 3:32 AM

Hi @bitman

I was testing the script and for some reason I don't know, the transcription process just stops on its own after a few minutes. For example a 15 minute video always stops at 23%-24%. Have you ever faced this problem? What can I do to resolve it?

bitman wrote on 1/15/2024, 9:22 AM

@Vegas_Pro_Brasil I do not really use speech to text (I do use a lot of text to speech), the whisper thing was just a fun project I wanted to try out, so I do not recall I tried any longer audio than a few minutes. So It may have always been an issue, I am not sure. Anyway I think it is aways safer to cut stuff (video or audio) in smaller chunks which are more manageable and easier on your system to process.

On the other hand, there may be some internal script process "safety" timing I do not know off in the Vegas scripting engine which may time out if the script engine is idling too long whilst whisper is busy processing. If that is the case, the script may need extra code to keep itself busy!

Something @jetdv can maybe shed a light (if scripts can time out when idling).

jetdv wrote on 1/15/2024, 10:41 AM

The script, itself, should have no issue. However, newer versions of Vegas have started including something that checks if there's no responses so Vegas can properly shut down if it gets stuck so that might be coming into play depending on the version used.

Kilo wrote on 5/17/2024, 9:48 PM

Hello! I'm hoping you can do me a huge favor. I followed your tutorial on how to do auto subtitles in Vegas pro without the 365 service and after many hours of trial and error (my fault) I got it to work. Now I am seeing another problem which I never expected unfortunately! I make vertical videos and the subtitles I create are only 1-3 words long not entire sentences like whisper AI outputs. I've seen some people online suggesting changing the source code but I'm not sure I can do that since I downloaded it from the console like your tutorial showed. Do you have any suggestions on how i should fix this problem. Thanks a lot for any help you can give me i know your times valuable so i truly appreciate it

bitman wrote on 5/18/2024, 2:43 AM

@Kilo if you take the latest script (Whisper Speech To Text v7) and open it with notepad++ then go to line 669, there is a statement to check for spaces between words to determine the number of words before a line break. It is currently at value 9, meaning 9 words, which seems optimal for English in normal horizontal aspect. For vertical videos you can perhaps lower this value to 2 or 3... Feel free to experiment!

Kilo wrote on 5/18/2024, 3:59 PM

Thank you so much bitman, I know you don't have to help but just know this is a tremendous time saver for me and my business. Unfortunately I can not get it to work. when I change line 669 of the code it seems to make the words go below each other

kind of

like this

but i was hoping that I would have 2-3 words per text media generator on Vegas pro. I'm not sure if this is possible but if you have any suggestions I would be forever grateful! Thanks again!

Kilo wrote on 5/20/2024, 3:00 PM

Ive spent days trying to figure this out and have made almost no progress. I need to find a way to shorten the amount of words to 2-3 per vegas pro text section! If anyone can help in any way or even just suggest a possible solution i would do ANYTHING! ILL FOR REAL PAY SOMEONE! pls help <3

bitman wrote on 5/21/2024, 1:07 PM

@Kilo To limit the number of words without newline breaks, then the whisper generated .SRT internal contend needs to be split (including timestamps) before import. This can be achieved by extra new code in the script, or separately by an external application.

This could be a nice project, but I do not have the time to do it now. It is more like a winter project!

If you take the latest script (Whisper Speech To Text v7) and open it with notepad++ then go to line 295, you can change the font and lower the font size so you can cram more words on your vertical videos... Not really a solution, but I suppose all small bits help!

Kilo wrote on 5/22/2024, 1:42 AM

After awhile of messing around i found that if you add

"sw.WriteLine("whisper " + "\"" + myFile + "\"" + " --word_timestamps True --max_line_width 16 --max_line_count 1 " + modelOption);"

On line 162 instead of what is currently on 162 in V7 it will create srt. files that has sections of no more than 16 characters (or whatever you change that number to)

Thank you to bitman again, if you ever do continue this project i hope this helps u slightly. I may continue to improve this script to add a few other easy customizations in the future and i will post here with the results if i have any positive ones

bitman wrote on 5/22/2024, 1:02 PM

@Kilo Thanks for the tip, I suppose vertical videos as a final delivery are getting more popular for smartphone end use!

As a side note, it is a PIA to camouflage and integrate vertical video into 'normal' horizontal FHD aspect in a more or less visually satisfying fashion.

bitman wrote on 6/28/2024, 8:11 AM

Some new Vegas 21 builds (e.g build 315) may have issues with my script v7, I have a new version available v8 which checks on the Vegas main version only rather than main version + build version, otherwise it gets confused with Sony :)

here is the link to v8

https://www.dropbox.com/scl/fi/voojqanq3cz0a7le31e2r/Whisper-Speech-To-Text-v8.cs?rlkey=xeq7v2zlqk8ir0aq154ewv0sw&dl=0