Automatically place Markers on music beats

Comments

Gary James wrote on 3/12/2014, 5:34 PM
Erni, this is a Vegas Script, not an Extension. The script appears under the Vegas Tools / Scripting menu. If you don't see it there, try running the Rescan Script Menu Folder menu selection at the top of the drop down menu.
Erni wrote on 3/12/2014, 10:30 PM
Gary,

I use Windows and Vegas in spanish versions. This makes my error.

But is not here in Vegas 12.
I can see the Script it in Vegas 10.
And yes, I make the rescan many times.

I use an SSD disc for programs and another disc non SSD for documents and so. Maybe this is the mother of the problem. But not for Vegas 10. Mistery.

Thanks again.

Erni
Gary James wrote on 3/12/2014, 11:44 PM
Erni, have you tried Opening the script and Running it from the Vegas Run Script menu selection? Select Tools / Scripting and select the top menu item - Run Script.



When the Run Script Dialog opens up, go to your Documents Folder, then open the Vegas Script Menu folder and select the ImportMarkers DLL.

Grazie wrote on 3/13/2014, 3:18 AM
Gary, I'm having a BALL with your script.

I've been tweaking my output by using TWO settings of markers. One at 180Hz and another at 261Hz. Both with Bandwith set at 10HzThis has produced controllable and organic variation when I import BOTH sets of Markers to the Timeline. And that, right there, is a glimpse into to what you could do: Slicker Freq sliders that allow me to dial-in slide-in the audio freqs on the fly.

Question: Would you be able to make the process a bit more streamlined? Somehow to get the Playlist Tools to be more integrated in the VP Timeline experience?

I do wish you and jr would work closely on this one . . . . Well, I can hope!

Cheers

Grazie



PhillB wrote on 3/13/2014, 5:30 AM
Gary,
Tried importing the new file using US with vegas TL set to seconds and get this error. https://dl.dropboxusercontent.com/u/8579129/marker-file-using-US.png


This was the file I was trying to import.
https://dl.dropboxusercontent.com/u/8579129/Marker%20export%20using%20PLT.txt?dl=1
Gary James wrote on 3/13/2014, 8:35 AM
Phill,

Ok, I had hoped that U-S would just ignore the fact that the label field in the file was missing, but I guess not. For now you'll just have to continue using my ImportMarkers script.
PhillB wrote on 3/13/2014, 10:59 AM
Thanks Gary that's no problem.
Any further development of this product would be really appreciated, being able to produce markers in this way opens up a whole new area in editing stills & video.

Would it ever be possible to see real time when beats are being detected/exported as you play a music track so that you could tune the controls on the fly ?
Gary James wrote on 3/13/2014, 3:46 PM
Grazie, the addition of Beat Detection to Playlist Tools was based on a couple of reasons; It was a simple stand-alone program that had little other baggage tied to it; and It could be compiled as a 32 bit program, making it easier to integrate the BASS Audio Library. The docs for the BASS Library claim that with some additional tweaking, it can work with 64 bit or mixed mode programs. But my efforts to make this happen have not met any success. And unless I can get it to work with both 32 and 64 bit code, I can't integrate it into SVP either as a new feature in Timeline Tools, or as a stand-alone script.

When you say "Slicker Freq sliders that allow me to dial-in slide-in the audio freqs on the fly." I'm not exactly sure what this means. Are you asking for a real-time beat analysis capability? If so, the library code doesn't support this. The audio file must be loaded into memory then analyzed by the library code.

One more thing. "I do wish you and jr would work closely on this one." Who is JR?
Grazie wrote on 3/13/2014, 5:02 PM
Yes, you understood me.

And "jr", is Mr John Rofrano aka here as "JohnnyRoy" - the genius behind Ultimate S Pro!

Luvin' your script!

G

Gary James wrote on 3/13/2014, 11:38 PM
Ok, after some head scratching and a post to the Bass Audio Library forum, I've got it figured out how to get the Bass library to work in a mixed mode 32 / 64 bit application. The problem was in the convoluted way the Bass Library is presented and downloaded on the web site. There are plane vanilla installs for people working on Windows, Linux, and MAC OS's. And within the Windows OS there are several different .ZIP files containing the code and DLL's for C / C++, VB, Delphi, Assembly language, and .NET. But for whatever reason, the .Net installer package did not include the 64 bit libraries. I had to download and install them separately.

At this point I'll proceed with making a more seamless integrated presence in SVP. I'll take any suggestions on how this should look and feel. Scripts are OK, but they're Modal so you can't do anything in Vegas until the Script is closed. I'd rather have a window up that can stay open and still be able to run Vegas. So that means an Extension instead of a script. I'm also thinking about the possibility of saving and loading analyzer settings. Again, any comments are welcome.

Oh, and lastly I'd be more than happy to talk with JR or anyone else interested in integrating this capability into their code. 99.99% of what it takes to do this is built into the Bass Audio Library. The challenge was sifting through the literally thousands of API calls, Data structures, and Object Classes that come with the Library, My code just uses what's already in there. For a commercial software house the big deal is paying the €950 / $1,317 licensing fee to use the library in one commercial product.
Grazie wrote on 3/14/2014, 2:14 AM
Gary, I have to admit, that your first paragraph, although impressed, went some distance above my head, but, I'm understanding the intricacies that you and pro programmers need to wrangle to get what is needed to do. That I get!

OK, on this Forum we have a bunch of iZotope RX Audio Restoration s/w users. One of the valuable functions iZoRX does is to "Learn" the issues that it is being presented with. Be it background "mush", or "clicks" and "crackles" on scratched Vinyl, it firstly needs these to register them and then to be able to remove them. I see/hear that your approach is somewhat similar, in as much as your script/library(?) can register the Beats and have it "Learn" where they appear in the WAV - or whatever... Now, I was thinking, and maybe Phil was to, that having a neat Vegas GUI that would allow the User to have the ability for the Script to be able to "Learn" the Beats would be what we'd be wanting to achieve.

So, here's how I see this working:

1] Select/click on Vegas Audio Event - and here, for me, that would have been my ZZ-Top-ish track.

2] Run Script ( . . or maybe a fully fledged GUI - hence your collaboration with John Rofrano? John also has that all important understand in the music and audio too!) that would allow several frequency sweeps or "passes" to produce that all important "Profile" that could be "Learnt".

3] Click on "Create" markers . .. and eh Voila!

Now, I can think of many more tweaks and GUI designs that would make this a "seamless" and maybe inspiring process, but in essence that's how I see it working.

Oh yeah other Tweaks!

A] Bass beats to have a Colour Patch VFX

B] A selected Freq that has an VFX applied to that marker position.

D] Transitionals on the Beat

E] NewBlue Pro Titler JUMPS in Text related to the Beats.

F] And the BIG one : Select Event/s on a certain beat!!! - Waoh.....

Gary, SCS must be reading this was much warmth, as this type of usage of VEGAS<>Scripts<>Collaboration must be what they've been after.

If you wish to setup a Face-To-Face through a SKYPE session then I'd be more than happy spend time with you.

Best wishes,

Grazie
Gary James wrote on 3/24/2014, 11:21 AM
Whew ... what a list. Ok, I didn't get to many of the items mentioned, because so many of them are outside my programming comfort zone, But what I have done in the past several days is bundle up many of these feature requests into a new -yet to be released- version of Timeline Tools. I've added a new Audio Tools Tab where you specify your audio track, choose your audio event(s), and configure and start the analyzer. On completion, the beat markers are automatically laid down on the Vegas Timeline. If they don't look right, you can click the Vegas Undo button, make adjustments to the analyzer and try again. If things look good, you should browse through the Markers and manually make any needed tweaks. From there Click on the Track Tools and Options Tab and select the new "Align Events to Markers" Radio Button selection. When ready click the Adjust Events button. What this does is start with the first Marker on the Timeline and move the first targeted video Event to that position. Then the video Event Length is adjusted to fit between the fist and second marker. This process is repeated for all remaining Markers and targeted Events.

With the right choice of Audio sound track containing prominent beats, Analyzer settings, some manual tweaking to add or remove extraneous Markers, and a logical progression of still images, you should be able to create a rapid fire video slideshow synced to the beat of your music, in just a small fraction of the time it would have once required.

Click on this link to download the latest Unreleased Timeline Tools containing all these new features. As before, I don't have a help file ready yet, so you'll have to boldly go where no man has has gone before.

I'd appreciate any feedback you'd like to make on this latest effort.
Erni wrote on 3/24/2014, 12:15 PM
Gary, works fine now. Many thanks for you effort!! Good man!

Erni
FocusOnVegas wrote on 3/24/2014, 3:42 PM
Thank you for sharing your updated Timeline Tools with beat detection.
This is a strong feature Vegas Pro is missing!

For audio with less prominent beats there is another method I use :
1. Let a bpm analyzer like MixMeister (free) analyze the beats per minutes.
2. Choose in Vegas under " Project Properties" , " Ruler" , "Ruler Time Format" : "Measures & Beats" and under "Beats per minute" the result of the analysis (like 115,8 bpm).
3. Load your audio file in Vegas.
4. Place the cursor at the start of the first beat in the audio file.
5. Start VASST Marker Maker app (not free but cheap, 10 dollar).
6. Create markers on every 4 (or other number) beats.
7. When the timing (at the end) is out of sync : remove the markers selectively with Marker Maker and repeat the process.

Wouldn't it be nice if we could combine these two methods :
1. The Timeline Tools audio analyzer analysis the bpm of the audio automatically or the user provides the bpm manually.
2. The user provides the number of beats for every marker (f.i. every 4 beats a marker).
3. The Timeline Tools audio analyzer analysis the audio.
4. Timeline tools corrects the potential markers with the bpm information (the distance between markers should be more or less the same; 2 or more markers at a short distance is not possible).
5. Optional : Timeline tools analysis the audio automatically a second time with different parameters to enhance the reliability of the placing of the markers.
6. Timeline Tools places the markers on the Vegas timeline.

It maybe looks a little bit ambitious, but, except the automatically analyzing the beats per minute, most of these steps can be done by manipultating marker data (array or with marker text files).

I hope you are tempted to experiment with some of the ideas above......

A last note. I got an error when I tried to use the Timeline audio analyzer a second time. I think it is related to the audio library you use (maybe the library is opened twice before closing or something like that; Windows 7 and Vegas Pro 64 bit).

Keep going with your good work!
JJKizak wrote on 3/24/2014, 4:03 PM
Messing with adding auto markers to drum beats in Acid Pro a while back to a recorded track I did find out how good the drummer wasn't.
JJK
Gary James wrote on 3/24/2014, 6:05 PM
"I got an error when I tried to use the Timeline audio analyzer a second time."

Strange. I ran this build many times using both Vegas 9, 32 bit and Vegas 11, 64 bit without problems. I'm also running Win 7 Pro. Can you duplicate the error? And if so, can you document each step you take to reproduce the problem?

Not being a musician, all the things you mentioned about working in beats per minute on the Timeline is pretty foreign to me. I'll have to take a crash course (bad terminology for both Aircraft and Software) on understanding the concepts. Until then, if you have a chance to use what's currently there, I'd like your opinion on usability, and convenience. Thanks.
FocusOnVegas wrote on 3/25/2014, 1:45 PM
Gary, the (first) error is :
" Error detected opening Audio file : BASS_ERROR_FILEFORM".

Steps to reproduce (Windows 7, 64 bit, Vegas Pro 12, 64 bit):
1. Select audio track and event with audio. Place cursor at the start of the audio.
2. Open Timeline tools, choose tab " Audio Tools"
3. Select the track to analyze.
4. Press " Process audio events"
5. Normal results.
6. Close Timeline tools (window).
7. Open Timeline tools, choose tab " Audio Tools" --> error above (first error)
8. Close Timeline tools.
9. Open TImeline tools. New error : Header window = " Unexpected Error Initializing Timeline Tools : step2". Content window : " Attempted to read or write protected memory. This often an indication that other memory is corrupt".

Regarding my remarks : I am happy what you already have done with beat detection. But I think the workflow and user experience can be further enhanced. For me the average length for clips in a montage with music is 1 - 5 sec. So I don't use every marker of every detected beat. In case the audio has a rhythm of 120 beat per minute (120 bpm), there will probably be beats and markers every 0,5 sec. I will use f.i. only every 4 th marker. If it is possible in Timeline tools to choose to place a marker on every beat, every 2 beats, every 3 beats etc., there will be needed less manual proceedings.

The bpm of a audio file could be used to enhance the reliability of the beat detection. If the bpm = 120 and there are 3 markers (detections) within 0,5 sec., it is likely one marker is "wrong" and can be corrected.

If you are able to extend Timeline tools with an option to choose the number of detected beats for a marker, it will be great........



Gary James wrote on 3/26/2014, 4:57 PM
Ok, I reviewed your steps to reproduce the problem and found the cause of the problem. It turns out the design of the BASS Audio Library is not particularly well suited for use in a Vegas Extension. For you programmers, the library is based on several standard C DLL files accessed through a .Net wrapper DLL. When I tested the Beat Detection feature I never closed Timeline Tools between running my multiple passes. And that's why it worked. The way Windows works internally is that once a DLL is loaded by a Vegas Extension, it never gets unloaded until the main application is closed - in this case Sony Vegas itself. This is true even if the Vegas Extension is unloaded first So each time you reopened Timeline Tools, I was reinitializing the BASS DLLs, that were never unloaded when Timeline Tools was closed. And this caused an internal memory fault in the BASS Library. Their code design works fine for an application that's started, then run till completion and eventually closed. But when it's used in a Plug-In or Extension, it has some squirrely issues.

I believe I have a work-around for this. It's a bit screwy. But, whatever works - as they say.

Funny you should mention a way to decimate the number of Markers created on the Timeline. I had thought about that, then got distracted by several interruptions, and never went back to it. I think I have a way of doing this that can leverage off some other existing features in Timeline Tools. I'll get that added in to TLT when I get my other work-around working 100%.
FocusOnVegas wrote on 3/27/2014, 1:47 PM
Gary, in case you can not solve the problems with the BASS Audio Library, maybe this library is an altenative :

http://naudio.codeplex.com/

It is free (Microsoft Public License) and seems to be a (native?) .NET library.
I have no experience with it an I don't know if it has all features (filters) you need.

But in case..............
Gary James wrote on 3/28/2014, 5:36 PM
Alright, I've updated Timeline Tools and placed it up on my Dropbox folder. This version includes the work-around that should let you run the program following the same steps that previously caused an error. It also includes a new feature that can decimate the number of Markers returned by the beat analyzer.

In this version the Audio Beat Analyzer works just as it did before. You tweak your settings and it returns gobs of beat Markers. To Decimate these Markers, click on the Markers, Regions & FX tab, then to the right of the Marker list, there's a button labeled Decimate. Click on this button and the Decimater tool window opens up. Choose your decimation factor, and select a starting Marker if you want it to be anything other than the first one. A decimation factor of 2 will leave 1 out of every 2 Markers untouched. A factor of 3 leaves 1 out of every 3 Markers untouched, and so on.

Click on the Apply button, and you'll see the highlighted markers in the Marker list that will be deleted. Finally, click on the Delete button just above the Decimate button to perform the actual deletion of Markers.

Download New Update

Let me know if my work-around fixed the problem you had previously seen. Thanks.
FocusOnVegas wrote on 3/29/2014, 4:38 AM
Thank you for your update, the error seems to be gone; it is possible to close and open Timeline tools sevaral times without an error message.

The decimate function is a good addition to limit " unwanted" markers.

With the " Align to markers" function semi automatic aligning of events at markers is possible.

I ' ll give you more feedback when I have used the new functions a couple of times in a "real" workflow.

Thank you for sharing Timeline Tools.


FocusOnVegas wrote on 4/2/2014, 4:02 PM
Gary, I am experimenting with a good (scientific) beat detection algorithm, see :
http://www.eecs.qmul.ac.uk/~simond/beatroot/ (stand alone program)
or
https://code.soundsoftware.ac.uk/projects/beatroot-vamp/files (plugin for Audacity).

I want to convert the exported results of this beat detection algorithm in a Timeline Tools compatible marker file format (.txt) with a small program.

I think there is a little bug with the marker writing / reading of Timeline tools.

Can you try these steps in Timeline Tools :
1. Start a Vegas project and choose as ruler time format "Seconds".
2. Load an audio file and place some markers with and without names.
3. Open Timeline Tools, TAB " Markers, Regions &FX".
4. Choose as Marker format "Timecode".
5. Select all the markers.
6. "Export Selected Markers & Regions".
7. Delete the markers.
8. "Import Markers & Regions" and choose the exported marker textfile.
Result : The markers are not loaded.
In the marker textfile I can find the marker positions like 15,560, 45,520.
The marker names are not present in the file and the markers don't load in Timeline Tools.

Can you confirm this is a bug in Timeline tools?
Correct me if I am wrong.

Gary James wrote on 4/2/2014, 10:07 PM
"In the marker textfile I can find the marker positions like 15,560, 45,520. The marker names are not present in the file and the markers don't load in Timeline Tools."

I tried out your steps as given. The reason you don't have any marker names could simply be that you didn't check the Include Labels checkbox when you exported your markers. Checking this option adds Marker labels along with the marker timecode values.

You stated you can find marker positions like 15,560, 45,520. Are these values all on a single line in the file, or each value is on its own line? Timeline Tools does not support Importing Markers & Labels if the file was written as a single line of text in the file. There's an option checkbox Write one line file, that controls if a one line file, or multiple row file is written. Make sure this is NOT checked. Does this help?
FocusOnVegas wrote on 4/3/2014, 11:49 AM
Gary,

The problems are with the marker format "Timecode".

I can save and load successfully markers in the "Framenumber" or "Time" format.
The same marker saved as "Timecode" format will not load.
The file seems to be correct : a marker with no name at framenumber 1743 is presented as " 69,720:"

There seems to be a bug in reading the "Timecode" marker format.

Update : The problematic combination seems to be : Ruler time format in Vegas = "Seconds" and the marker format in Timeline Tools = " Timecode" .