SCS TO OFFER OPTICAL FLOW...?

Jay Gladwell wrote on 7/28/2009, 6:42 AM

Now that I got your attention...

There is a current thread bemoaning the lack of certain capabilities in Vegas 9.0. It's not the first, and I doubt it will be the last.

Instead of posting rude rants, why don't we take advantage of the Product Suggestion page and politely ask SCS to include whatever we think would help make Vegas a better program?

For example, after reading the thread about slow motion and seeing example's of Optical Flow in Apple's Shake, I went to the Product Suggestion page and asked that they consider something comparable be added to Vegas. If enough of us do this--showing it's something the users want--perhaps they will include it at some future date. Even if it's offered as a plug-in, like Noise reduction, that's better than not having it at all.

The Product Suggestion page is our friend! Let's use it.


Comments

apit34356 wrote on 7/28/2009, 6:58 AM
Gee, be helpful.......... but that evil parasite eating my brain cells demands that I stomp around and moan about SCS and not the source of the problems... ;-)

discussion of problems is a good thing, it helps promote solutions. But the whining about SCS being clueless is really a waste of everyone's time, including Blinks--- ;-) (couldn't pass on that Blink...... its that evil parasite..... it's been on a rampage every since Canada won't import Madcow infected meat). ;-) It's union break time now, time to go and kick around some homeless people (FCP users), the west coast is so much fun to visit, golf clubs and rain gear are really versa tools. ;-)
farss wrote on 7/28/2009, 7:08 AM
Not to rain on anyones parade however I think they'd already know we want it and many other things as well such as image stabilisation. They all come back to the same problem.
What I don't know is how much else of what makes Vegas what it is would have to be broken in order for us to get what we want. I'd love for there to be a frank and open discussion about this, it's a significant issue that needs to be dealt with and sooner rather than later. The competition is not standing still and they're not hampered by dogma.

Bob.
Jay Gladwell wrote on 7/28/2009, 7:20 AM

"... I think they'd already know we want it and many other things as well such as image stabilisation."

Bob, you may be right, but I've learned not to assume anything. As several have pointed out before, this is a "user forum" first. I'd rather not assume that SCS wades through every post here to see what we want. Hence the push for using the Product Suggestion page.

The remainder of your post I understand and agree with. I'm not altogether sure it fits this particular topic. Fixes are one thing, new capabilities are another. Both are needed!


Coursedesign wrote on 7/28/2009, 9:54 AM
Vegas would have had optical flow and many other useful features a long time ago, if only SF/SCS hadn't been so willingly herded into MS's walled garden of old APIs.

MS represented a long time ago that they understood professional video; the reality has been that professional NLEs have had to find their own ways to implement key functionality.

QuickTime has had the goods for a long time, but MS has fought tooth and nail to keep it crippled in Windows; QT is slow in Vegas because it was never a priority for SCS.

Boris could offer optical flow in Vegas immediately if they were given an API that could look at multiple frames.

I suspect that SCS is caught between a rock and a hard place right now, with 64-bit APIs becoming necessary while VFW is the laughing stock of the industry.

Let's remember that Vegas started as a studio audio app that added "multimedia" (320x240 frames from CD-ROM or, if you were lucky, downloads), then "DV25" and over time other formats. I don't think anyone thought that it would one day be used for large, complex projects...

FCP was also a "DV NLE" that has since seen use in many Hollywood movies only because of the efforts of Walter Murch. Walter was pissed off with Avid's attitude at the time, and wanted to edit large production films in FCP, which was not practical at the time. He kept up the pressure with help from 3rd party vendors, and Apple eventually relented and put in the features that were needed, except for Avid class media management. Walter got around that by writing his own custom media manager in Filemaker (a truly great app development environment) that interfaced with FCP.

I would hope that this group can do the same: keep up the pressure on SCS to add better APIs so 3rd party developers can add features that may not be cost-effective for SCS to develop in-house!
blink3times wrote on 7/28/2009, 10:18 AM
"discussion of problems is a good thing, it helps promote solutions. But the whining about SCS being clueless is really a waste of everyone's time, including Blinks--- ;-) (couldn't pass on that Blink...... its that evil parasite..... it's been on a rampage every since Canada won't import Madcow infected meat). ;-) It's union break time now, time to go and kick around some homeless people (FCP users), the west coast is so much fun to visit, golf clubs and rain gear are really versa tools. ;-)"

Actually it's Swine Flu now. The poor bloody pig farmers are taking a pounding this time.

But yeah... I totally agree with this thread. Getting on these forums with "scs sucks" and other such rubbish is absolutely counter productive and serves no other purpose than to piss off people that have nothing to do with the problems you're having. There's a suggestion box, troubleshooting guides, problem ticket options.... many different avenues that are far more productive than to come on the forums with such trash as "this sucks".

People on this site are also quite helpful.... IF.... approached in somewhat of a grown up manor..... shall we say.
farss wrote on 7/28/2009, 3:51 PM
"QuickTime has had the goods for a long time"

How so?

Optical flow processing has been available on the Windows platform for a long time, well before it was available in any Apple product. If you wanted it under OSX AE was the only way to get it.

QT to the best of my knowledge is still unable to suppport active metadata, something that's been available under Windows for years.

"Let's remember that Vegas started as a studio audio app"

That's very likely very much the source of the underlying issue.
My best guess is it's ability to go forward as an audio app was hamstrung once video was added. It's ability to go forward as a video app is hamstrung by it's audio heritage.

Bob.
DJPadre wrote on 7/28/2009, 4:50 PM
Wow, some of you guys really are a delicate bunch, which is fair enough considering the nature of this beast.
There will always be those who dont beleive ones right to speak ones mind as they se fit is a viable means to actively communicate on a variety of levels.

Yes what we do has its serious side, but being such a personal choice (NLE's) , whether or not one believes something "sucks" is a right of choice to express oneself at that time.

Whether it offends is not the point. This isnt a fluffed up group so pretentious as to not even bother to listen and the fact that inital post DID get the attention, the point has been proven to be made.

Suggestions are all well and good. Been there done that... The issue with Vegas itself, is not optical flow or wow factor effects. The issue Vegas has is the fact it cannot manage Progressive scan footage from the get go.

Whether or not this app was born from an audio DAW, is also not the point or the issue.
Those days are long gone .
The point Im making here is that If SCS decide to push Vegas as a Video Editing App, then SCS must adapt to the deal with the needs of todays formats and users.
Consider DVCProHD100... vegas STILL doesnt support that format, and no, its not fair that vegas users have to create workarounds to deal with this format. When you consider TV stations will only accept XDCam or DVCProHD, this lack of support is appalling.
Whether or not its in the API or licensing issues with Pana again is not the point. These issues CAN be dealt with, if politics didnt get in the way.
At teh end of the day, there are users out there who are automatically cut off from this application simply becuase of their camera of choice.

Vegas is a great tool, however what Sony and their Beta testers need to do is consider the fact that not everyone usees Sony camera HW. Not everyone chooses to shoot interlaced and not everyone uses Sony patented codecs.

SCS need to consider that the world will NOT adapt to suit them. SCS must adapt to the world...,
Its that simple

johnmeyer wrote on 7/28/2009, 5:20 PM
I do not think it likely that SCS will ever pursue this. Many years ago, when I briefly had the ear of a few developers in Madison, I sent a long paper to them providing advice on how they could win in the marketplace. They never took any of the advice. I copy below a very short portion of what I sent to them almost four years ago.
----------------------------
Thursday, September 22, 2005 10:41 AM

I have spent the last eight years consulting with various startups, although for family reasons I have had to stay at home the past three years (hence my constant presence on the Sony forums).

The preview of what I would like to recommend is this:

Long term, to win in this market, you need to invest in underlying technology that will help differentiate your product. You also need to invest in relationships that will help fill the gaps in the product and its support that even a company of Sony's size can't fill.

On the technology side, the first area I would invest in would be motion estimation. Yes, I know you have lots of work to complete the underlying plumbing needed to support the transition to HDV. However, until delivery (Blu-Ray, etc.) is widely available, this (HDV) is going to continue to be a numerically small market (although an influential one to be sure, since most of the press, and most of the media will be writing about this and using it themselves). However, event videographers and corporate video have relatively little use for this, especially given the considerable penalty in time required to complete post-production.

On the other hand, there are many things that still need to be done that would affect every current and potential Vegas user, whether HD or SD. I break these down into two main areas:

1. Features that would make someone switch (from another editing program to Vegas).
2. Features that would make an existing user stay.

It is a tall order to ask someone to abandon their editing paradigm. The feature or features must be pretty compelling. And, they must be difficult to duplicate and -- if possible -- patentable, so the competition can't easily catch up. I believe there is technology that might possibly produce such dividends. In particular, I have spent quite a bit of time working with various crude programs that use "motion estimation" in order to create intermediate frames. This technology is, of course, directly related to much of the technology used in the pre-processing phase of MPEG-2 encoding. It provides a way of creating intermediate frames from information that is derived from both preceding and succeeding frames.

My recommendation is that Sony invest in improving this technology and then applying it to the following areas. Some of these applications, I believe, may be patentable. I should point out that I have no other involvements that keep me from making this recommendation, and I also hereby release any claim to the idea. You guys can have it.

Here are the three major applications of such a technology:

1. Fantastic slow motion. When I look at all the features in Vegas, I believe that 90% of them are probably used by 10% of your users. What's more, if you were to take a survey to verify this, and if you asked the right questions, I would further bet that you'd find that if you look at PROJECTS instead of USERS, that about 99% of the projects created don't use most of the features (like the jillillions of transitions, 3D compositing, etc.). However, I bet you'd find that a very large percentage of users employ some sort of speed change in their videos. In particular, slow motion is probably the most common special effect used in video. However, as good as the Vegas slow motion facility has become, it cannot hold a candle to what is possible with motion estimation. I have used several low-end and free products (MotionPerfect from Dynapel, and mVTools which is a plugin for AVISynth). Neither of these products has ever been enhanced since the initial release, and there are lots of problems with them. However, on certain types of video, the results are awe inspiring.

Fantastic slow motion is a reason to switch.

2. Bad/missing frame correction. This is the feature I think you could patent. I also think it is one of the most compelling reasons to switch. Imagine that you've got a single bad frame (or a missing frame due to a dropped frame on capture). This could also be due to a noise glitch on the tape (those annoying white pixels sprinkled across the frame). It can even be in the video itself (a strobe from a person taking a flash picture, which you'd like to remove). With motion estimation, you can synthesize the missing frame. With the proper UI, this could be turned into a one-button fix. Click: bad frame gone. Can you imagine the advertising and marketing possibilities with such a feature? You could even go further by letting the user mask the bad portions of the frame (for pixel dropouts) and then only replace those bad sections.

3. Low frame rate source material. Even as the main market for Vegas moves inexorably (and slowly) towards HD, there is an even bigger and faster move in the other direction. Still image cameras often have limited movie capabilities. Some cell phones can take video. Also, videoconferencing continues its decade long move towards acceptance (I can't believe that it has been thirteen years since I ran ShareVision, which was one of the pioneer desktop videoconferencing companies). The point is, there is a lot of low-frame-rate video out there. While no serious videographer is going to intentionally shoot or capture video using any of this technology, in some cases this is all that is going to be available. In news, we see this footage used all the time. More important to your market, in weddings, the bride and groom may have video from guests that they would like incorporated. Corporate video often includes contributions from employees who have captured something at a trade show or other event, using their personal still camera or cell phone. With motion estimation, such 10-15 fps video can be changed to 25 or 29.97 by synthesizing and inserting additional frames. I just did this a few days ago with some sports video taken with a 10 fps Canon still camera. I tried using Vegas (using resample), but the results were not good. I then tried MotionPerfect, and on some shots the results were spectacular. On others, the artifacts from the flaws in their design were pretty overwhelming (i.e., the results were not good). I firmly believe that these flaws are almost entirely in their implementation, and not in the underlying concept: MPEG-2 wouldn't look as good as it does if it weren't possible to create intermediate frames.


Jay Gladwell wrote on 7/28/2009, 7:22 PM

"It is a tall order to ask someone to abandon their editing paradigm. The feature or features must be pretty compelling.

Believe it or not, John, that very thought (maybe not in those exact words) went through my mind as I looked at Shake and it's Optical Flow (and other capabilities).

But in all fairness, the thought that immediately followed was the outrageous cost that it would require to switch to a Mac editing system (along with losing the ease of editing in Vegas). That was when I decided to "suggest" to SCS to come up with an alternative.

However, your point is still valid.


farss wrote on 7/28/2009, 7:35 PM
I'd add another point.
Frame rate conversion. Specifically between Region50 and Region60.

Bob.
John_Cline wrote on 7/28/2009, 7:48 PM
If I want decent slow motion, I shoot overcranked on an EX1. Otherwise, I use Twixtor or some of the other third-party software designed specifically for this task. I also don't have much use for image stabilization as I take great care to NOT shoot unstable footage in the first place. Vegas covers about 98% of what I need to do, I have other ways to accomplish the other 2%. Adding Image stabilization and optical flow slow motion to satisfy the needs of a very few Vegas users would just add to the price of Vegas for everyone.

That said, anyone trying to do perfect slow motion from 24p source footage is expecting miracles.
Rory Cooper wrote on 7/28/2009, 10:01 PM
That said, anyone trying to do perfect slow motion from 24p source footage is expecting miracles

the apostle Boris RED converted me
John_Cline wrote on 7/28/2009, 10:40 PM
I said "perfect slow motion" and although Boris RED is quite good, it is not perfect. One word; "overcrank."
apit34356 wrote on 7/28/2009, 10:51 PM
This is funny, it seems that lot of people think Sony(parent) is new to film, video, broadcast, global marketing, patents in encoding, etc. Johnmeyer, I usually find your advice given to this forum to help others useful, but then you probably have more experience a perpendicular app more than the new user.

Not to a jerk, Johnmeyer, but you need to do some real IP work, patent lookups; you will see that most of all your general suggestions are ready patented or under some form IP licensing agreements. Its very hard to "create" a new approach to these technologies without major new hardware designs that do not employ past or current IP materials.

With your HP work history,JohnMeyer, I though it be obvious that being sued for infringing on someone's IP for Sony's deep pockets controls their direction and at keeping the wolves(attorneys) at bay are high proprieties.

You should know that no "technology" company in the business world can "accept" a non requested paper on what to research, what to develop and how to market. Sony could or would be sue for using such advice without "agreements". But now you post this "paper on how run to Sony's development" on a public forum, claiming that you tried to show them the way! Now, the few who don't know the tech business world are thinking how dumb can SCS be!

Consulting is your business, then why in the world would did you post something in public forum that you knew, or should have known, could never be accepted. I ask you, what do you think the "big boys" in business suits are going to do when you seek to do business with them or in negotiations with them for a small start-up?

JohnMeyer, I think your a good guy and really want to help people, but playing with loaded guns can ruin one's professional career or the one's your advising.



Jay Gladwell wrote on 7/29/2009, 4:56 AM

PLEASE, let's get this thread back on track and keep it there, setting personal differences aside!

John [Cline], I agree with you, using an EX (overcrank) is a great way to start, but there are some occasions where I would like the slo-mo to be even slower and smoother. I've looked at (and tried) a couple of stand alone apps. (from Dynapel). However, that was back in my SD days. Don't how it would work with HD. I haven't tried it.

Regardless of whether I am right or wrong, I think the inclusion of an app like Shake's Optical Flow would be a great benefit to the vast majority of Vegas users.

My real point is (the point of this thread), regardless of what the individual editor may want, he should use the Product Suggestion. Then, he might consider posting that suggestion here in the forum in hopes that other editors who agree would follow suit.


John_Cline wrote on 7/29/2009, 5:04 AM
Jay,

If SCS would incorporate high-quality slow motion into Vegas, I would certainly use it. I just don't necessarily want to pay (much) more for it.
Jay Gladwell wrote on 7/29/2009, 5:11 AM

John, I understand. That's why I suggested it might be treated as an option, like Noise Reduction was.

If it could deliever the quality of slo-mo of Optical Flow, I'd be happy to pay for it. One job could do that!


John_Cline wrote on 7/29/2009, 5:34 AM
I know that SCS updated the SDK for Vegas with version 9. I wonder if they removed the restriction that it could only serve up one frame to a filter? Maybe optical flow is now a possibility.
DJPadre wrote on 7/29/2009, 6:05 AM
Re Dynapel "Don't how it would work with HD. I haven't tried it."

It works if you render out to Huffy YUV. Im testing with Sony YUV as this install of windows pukes whenever I view as thumbnails and theres a huffy codec in there.

As far as overcranking goes, heres the problem. Does one shoot their job in 1080p, and overcrank (which locked at 720p) then upscale to 1080p from that 720p source?
THATS the problem here... if there was a way to overcrank at 1080p, then overcranking would be a viable solution to this dilemma.

As it stands, one must lower their res to overcrank (HVX or EX) and the most difficult thing about this is that by overcranking, you're restricting your filming/shooting style.
The point - Who knows when or where one might choose to use slowmotion?
Not all shots, especially for event work like weddings, is planned and executed to a script or list of must have slowmotion shots.

Ramping is also non existant.
Lets say we're editing a wedding, and the groom goes in for the kiss and as he pulls the brides veil up and over, we want to slow that piece out as he goes in for the kiss...
So from realtime it ramps down to 50% speed as we want to emphasis the kiss.

To achieve this with overcrank, one must overcrank the entire scene and use a seperate audio recorder as audio will be lost. I dont need to tell anyone that this in itself is overkill for a wedding...

The other issue i guess is that one may try to convert 1080i to 1080p. Personally I think it looks horrid and my PS3 scales SD better than Vegas can interpolate these 1080 fields into frames. Id rather run Scaled footage from a PS3 as opposed to diong this as frankly it looks MUCH better. and the ps3 scaler is kinda sucky too might add...

The only in between solution at this time, is to shoot 1080i and scale down to 720p.
This opiton DOES look incredible and running this material on BD player/PS3 scales the materail up to 1080i on a 720p panel (or native 720p) , or 1080p on a 1080p panel. This scaling is inherant to the playback device.

In any case, 720p as efficient as it is, isnt a selling point as "True HD" 1080p is.

I agree with the comment in regard to Vegas performing 98% of tasks, but for people like myself who've been shooting and editing progressive scan footage now for wow, 7yrs, the lack of progress with this format is a huge let down.


Jay Gladwell wrote on 7/29/2009, 6:52 AM

DJ, all I can say is I'm not having the issues with progressive you seem to be having. I can shoot overcranked and use the Velocity Envelope to ramp up and down without any problems.

Insofar as knowing "when or where one might choose to use slowmotion" would not be an issue if one were using something like Optical Flow. Hence, the request to SCS for something comparable.

Have you made such a submission to SCS? The more people that request it, the more apt they are to include it (in some form or another).


Jay Gladwell wrote on 7/29/2009, 6:56 AM

"I know that SCS updated the SDK for Vegas with version 9. I wonder if they removed the restriction that it could only serve up one frame to a filter? Maybe optical flow is now a possibility."

Not being at technically savvy as you are, John, I cannot provide an "educated" reply to that. I do trust your insights and can only hope you are correct!


Coursedesign wrote on 7/29/2009, 7:54 AM
Optical flow is "ancient," but its initial use for video was in the 1960s in some very expensive Snell & Wilcox gear.

Apple's Advanced Technology Group developed key optical flow practical technology in the late 1980s and early 1990s.

The first plug-in for After Effects that offered optical flow however was ReelSmart MotionBlur, released in 1999 This product is still in widespread use, as it works great for either adding or removing motion blur.

After Effects was Mac-only for many years.

QT has a number of other useful capabilities, but these are not supported by Vegas (although some can be accessed via extra steps in QT Pro).

There is no unique capability in QT that's needed for optical flow. The only API feature that is a hard requirement afaik is access to multiple frames.

I suspect that there is already some pressure inside SCS to do an end run around ye olde Microsoft VFW APIs that the Vegas code is written for, in order to provide a modern plug-in API.

The After Effects plug-in API would be good, but Adobe wants serious money for anything above the AE 4.0 version.

That's why FCP offers, in addition to the old AE plug-in API they've had for ages, the new FxPlug API, their own modern design that is GPU-accelerated and more. Really nice functionality, high performance, and easy to write code for.

Jay Gladwell wrote on 7/29/2009, 8:34 AM

"Optical flow is "ancient,"..."

I don't care. It works, and that's all that matters, isn't it?

"That's why FCP offers, in addition to the old AE plug-in API they've had for ages, the new FxPlug API, their own modern design that is GPU-accelerated and more. Really nice functionality, high performance, and easy to write code for."

You're not proselytizing are you, Bjorn?


Laurence wrote on 7/29/2009, 8:53 AM
Two things:

1/ If you are working at 30p and want a slow motion shot, why not just shoot that shot at 60i so you can slow it down. A slowed down 60i shot rendered to 30p is going to look better than the best motion estimation algorithm.

2/ If you already shot the footage at 30p, you could just use a motion estimation plugin in VirtualDub. Here's http://wareseeker.com/Video/virtualdub-msu-motion-estimation-filter-1.0.zip/290880one[/link] I just found with a quick Google search. Heck, I'll bet the excellent JM deshaker script could be rewritten to use vdub with this plugin!