Script idea: Deshaker replacement

RichMacDonald wrote on 2/15/2005, 9:53 PM
Ok, here's an idea. Long story short, I've spent the day (re-)exploring the programmng options for plugins. I have a thread over in the video thread, but I'm switching to ths forum because its about scripts. If you don't want to use DirectX6 and C++, the Sony SDK is an uphill battle. 1.5 yrs ago, Sony said they would try and update the SDK to C# and more recent DirectX, but it hasn't happened. (Not a slam: Demand has been low and they've been busy.) Lots of forums posts of people getting their toes wet with C# then drowning. Not something you can get productive in a week or two. I've looked at this several times but the initial hurdle is too great so I always blow it off. Sounds like I'm not alone. Too bad.

Scripting is simpler in comparison, however afaik you can't develop keyframeable plugins and so forth. You can create and specify motion keyframes, though, right?

I've been thinking about plugins because I'm annoyed at deshaker for VirtualDub and want an alternative: I love deshaker, but it resamples the video even if you're only moving in X and Y, which softens the video.

It just occurred to me: A darn good way to implement deshaker for Vegas is not to recreate a second avi file. Instead, a tool could analyze the video file then use scripting to insert a keyframe that performed the correcting X and Y shift, Zooming and Rotation. IOW, one keyframe per frame :-! Pretty cool, huh? Beats the manual keyframing I know some of us have done :-)

In fact, its possible to use the existng VirtualDub deshaker to perform the correction calculations (it generates a text log file with all the corrections), then use that as the input to the script.

So here is roughly the idea:

1) Select your clip(s) and run the script.
2) For each clip, launch VirtualDub, load deshaker and run in pass 1 mode.
3) Read the Deshaker.log file. This is a text file giving an X, Y, Zoom and rotation for each frame. (Actually two values per frame for interlaced video.)
4) Fit these values to a "dampening curve" based on settings in the script. These are the X, Y, Zoom and Rotation values that correct for the shaking in the original clip.
5) For each frame in the clip, insert a keyframe with the calculated X, Y, Zoom and Rotation values.

Long term improvements might be:

a) Replace VirtualDub deshaker in step 2 wth your own video clip analysis tool.

b) Deshaking messes up the edges of the video. The above approach can either leave these edges empty or can zoom the video to fill the screen (with softening, of course). VirtualDub deshaker also allows you to blend in previous and future frames to fill in the empty spaces and it does a fine job.

c) The script needs a form to allow the user to specify various settings for the corrections, i.e., "smoothing factors" for the X, Y, Zoom and Rotation. You could also turn off certain corrections like Zoom and Rotation, e.g., if you were shooting from a tripod. Ultimately you'd want these values themselves to be keyframeable.

------------
The idea seems pretty straightforward to me. Pretty doable and pretty useful. The only serious bit of new coding is converting the calculated camera movements with dampening to the actual adjustments in the motion keyframe.

Comments

johnmeyer wrote on 2/16/2005, 11:24 AM
If you can script the pan/crop x/y values, then your idea should be pretty simple to implement. Will the result look better? Without resampling, I suspect that the video will have a somewhat "jumpy" look to it. Also, you will no longer have the edge correction, so the borders of your video will now be ragged (since, as you point out, zoom will require resampling, and you're right back where you started).
RichMacDonald wrote on 2/16/2005, 11:35 AM
>Without resampling, I suspect that the video will have a somewhat "jumpy" look to it.

That's a concern, but I figured with a resolution of 1 pixel, that's not much "jump". I don't have Vegas handy, but doesn't it allow fractional pixel adjustments? And that begs the question: "Does Vegas resample if the XY adjustment is an integer value?"

Lack of edge correction: Yup that is a limitation. And its something deshaker does extremely well.
RichMacDonald wrote on 2/16/2005, 4:40 PM
>"Does Vegas resample if the XY adjustment is an integer value?"

Oh boy. Yes it does :-(
johnmeyer wrote on 2/16/2005, 4:54 PM
Do you turn off "Smart Resample" for the event? I suspect that it will still resample, but you should at least give it a try.
RichMacDonald wrote on 2/16/2005, 6:46 PM
>Do you turn off "Smart Resample" for the event? I suspect that it will still resample, but you should at least give it a try.

Good idea, but no it still resamples.
RichMacDonald wrote on 2/16/2005, 9:18 PM
Thngs are getting a little confusing here: With an interlaced sample the result was definitely soft, however I'm looking at the result on a computer and not a TV. I just tried the same test with a still and the project in progressive mode and the result was perfect, i.e., Vegas shifted the clip without any loss in accuracy. I'll keep investigating.

Apologies for going back and forth between forums, but the script idea is dead until I can get this resolved and the current issue really belongs over in the Video forum. Thread is Track Motion causes resampling even when it doesn't need t.
RichMacDonald wrote on 2/24/2005, 1:15 PM
Just an update: I've abandoned the idea of implementing deshaker in Vegas. However, I've finally solved the scripting crashes in VirtualDub. This makes it possible to write a vegas script to calll VirtualDub, load deshaker, run the 2-pass on a clip including the appending and trimming required, then add this clip into vegas.

The VirtualDub script issue is described at virtualdub forum. It was simply a matter of rearranging some lines. Blind luck that solved it :-)

The Vegas script calls for:

1) Getting the selected clips. This is trivial.

2) For each clip, launch virtualdub+deshaker to render a new clip. One trick is that we need to convert the clip selection in vegas to a clip selection in virtualdub, because we might only need to deshake subsections of a clip. Also, we'll need to point the script to the virtualdub exe file, the deshaker pass1 script, and the pass2 script. So we need a simple dialog with file selection. Or we could just re-implement the deshaker options in the vegas script. Other issues include selecting which codec to use in pass 2.

3) Get that clip into vegas. This requires some options. Should we (a) overwrite the original clip, (b) add the new deshaken clip as a new take, or (c) add the deshaken clip to a new track? Or (d) make this an option in the vegas script?

(4) I sometimes like to run deshaker pass 1 on a special clip that I have masked to isolate the moving image I want to stabilize. Then I like to run pass 2 on the original unmasked clip. So this is another option.

I'm planning on messing around with this, as time permits. But I'm also planning on doing it in java, which is going to make it impossible for others to extend. Is there any interest in collaborating on this? I'd be happy to hand it over to an experienced script writer.
Laurence wrote on 5/20/2005, 4:56 PM
Now that would be extremely cool! I'd love to be able to do that.
Laurence wrote on 6/2/2005, 9:38 AM
Any progress on this?