Speech to Text via Whisper openAI

bitman wrote on 11/11/2022, 9:40 AM

Some good news, I have been diving into an alternative to "Vegas Pro Speech to Text". Although it is a very fine feature and easy to use, it has some drawbacks: first of all, it is a Vegas 365 - only feature (I hope this may change). Lots of people will not be able to use it - those without a subscription. Like all AI based stuff it is dependent on the model and results may vary. It also lacks a way to tune for quality and language. I suspect it also being more favorable to English.

So here is an alternative: Whisper openAI.

I have created a simple Vegas script to call whisper and convert speech to text. Just place the cursor over an event on the timeline and the script will create result files with text. In a future version I can extend this to create subtitles from these result files on the timeline, feel free to add this or add more of the whisper capabilities like quality, language and translation options. Refer to the document on Whisper at the bottom of this post.

Here is the link to the Vegas script called "Whisper Speech To Text":

https://www.dropbox.com/s/8gpb8w1fjj9bnt1/Whisper%20Speech%20To%20Text.cs?dl=0

Update 29/11/2022:

I made a new improved script with UI to select different transcode model options, a translate option, and a UI option to import the subtitles from the generated files to a new track

Here is the link to the new Vegas script called "Whisper Speech To Text v2":

https://www.dropbox.com/s/qxpaj4c8ybnlxgq/Whisper%20Speech%20To%20Text%20v2.cs?dl=0

The only caveat is that it requires quite a bit of effort to get whisper installed, it depends on Python, GIT, FFmpeg, etc. and setting of environment variables. So, you need to install a bunch of supporting stuff before you can use whisper. But it is doable. For this purpose, I have put together a document on how to use and install whisper (and its dependent programs), it has all the links to get you up and running.

Here is the link to the document on Whisper openAI:

https://www.dropbox.com/s/dh62ripb58xth86/AI%20whisper.docx?dl=0

Have fun!

Back to post

Comments

Todd-A0 wrote on 3/4/2023, 9:03 PM

Subtitle Edit is ready to go for these implementations so whenever we get a fixed or functional new one it can be added

It's great software, I hope if they make the change to GPU processing for whisper, it doesn't get the same problems as the GPU whisper versions.

Maybe run it by RX10 first, separate out the person you need, and use whisper next.

Great idea! 👍

RogerS wrote on 3/5/2023, 1:22 AM

Fora a non-python Whisper that does CPU or GPU you can grab it here: https://github.com/Purfview/whisper-standalone-win

It's not working in SubtitleEdit at the moment but works from the command prompt (run cmd as admin). It doesn't seem to have repeated lines.

Save it somewhere, dump ffmpeg.exe to the folder with whisper.exe, change the command prompt folder to there "cd C:\Whisper\" for example. Try this as a template (you can changer the language, location and model type).

whisper.exe --device cuda --language en --model "base" "C:\Videos\video name.mp4"

Todd-A0 wrote on 3/14/2023, 1:53 AM

This is the whisper variant i'm using currently https://github.com/Dadangdut33/Speech-Translate/releases/tag/1.1.0

It seems pretty good, in this example created subtitles for a 3min video using large dictionary in 1 minute (rtx 3080) . It gets things almost perfect until about 2 minutes where timing begins to be affected. I thought others interested in translation could use this as a barometer of sorts, and even download this video and compare the app version they're using. Russian historically has been difficult for whisper to do a good job at, If the whisper version you're using does a better job let us know

This uses GPU, maybe only Nvidia. It has no integration with any NLE.

Todd-A0 wrote on 3/27/2023, 11:31 PM

I tried the new version of StoryToolKit (Nvidia GPU only) by downloading the video in my last message and re-translating it. Top subs are the new translation. https://github.com/octimot/StoryToolkitAI/releases/tag/v0.17.16

It doesn't have the same timing problems seen with Speech-Translate, but as a negative it's formatting not as good, and instead of using multiple shorter sentences it seem to like to form paragraphs instead. Neither perfect options, but StoryToolKit in standalone mode (for Vegas users) possibly better choice, just need to break up the sub paragraphs manually where required

If your whisper translator does a better job, please share

wwaag wrote on 9/5/2023, 1:43 PM

Just wrote a new Batch WhisperAI Speech to Text tool and created a new thread. Here's the link https://www.vegascreativesoftware.info/us/forum/happyotter-batchwhisperai-speech-to-text--142423/