SD --> HD Upconverter plugin for Vegas?

2G wrote on 7/15/2011, 7:48 AM
One of my contract camera operator apparently doesn't yet know the difference between shooting in SD and shooting in HD. He delivered several hours of footage from a once-in-a-lifetime event in standard definition, and I'm producing it in HD. I can work around most of it without requiring his footage. But I really need his shots in a few situations. So I'm looking for an SD-->HD upconverter. I found a Red Giant plugin for After Effects for $99, which is fine. But I'd really like to do it in Vegas if I can find something comparable with a similar price.

Any recommendations for a Vegas upconverter?

Thanks.

Comments

Laurence wrote on 7/15/2011, 8:05 AM
Vegas will upconvert with no additional software using the bicubic resize algorithm. Boris Red or BCC7 can do an uprez using better math. For free (but working outside of Vegas) you could try VirtualDub and the Lancos3 resize. Is the SD footage interlaced or progressive? Most of the 3rd party uprez plugins are only designed to work with progressive.
craftech wrote on 7/15/2011, 2:39 PM
For free (but working outside of Vegas) you could try VirtualDub and the Lancos3 resize.

Actually according to the AviSynth Resize Wiki, Lanczos 4 Resize is preferrable for upconverting.

"For upsampling (making the image larger), the filter is sized such that the entire equation falls across 4 input samples, making it a 4-tap filter. It doesn't matter how big the output image is going to be - it's still just 4 taps.

To utilize it in a script one merely has to substitute Lanczos4 for Lanczos.

John Rofrano posted a link and a review for a Vegas Plugin in a similar thread.

John

Marco. wrote on 7/15/2011, 3:10 PM
There is also an UpRez plug-in for Vegas Pro available from Boris FX. It's part of Boris Continuum Complete but also available as single unit for 99,00 US-Dollar.
musicvid10 wrote on 7/15/2011, 7:36 PM
Lanczos 4 has some quirks from what I've read; and may not be quite ready for prime time.

Lanczos 3 is both time-tested and predictable.
Laurence wrote on 7/15/2011, 8:41 PM
If time, money and hard disk space are no object, you could make a png image sequence and batch resize the pngs with Photzoom Pro or Genuine Fractals. The uprezzing in Photozoom Pro can be absolutely stunning with clean source material.
PeterDuke wrote on 7/16/2011, 6:38 AM
If you are going to uprez a sequence of stills you should deinterlace the video first. What deinterlacer?
PeterDuke wrote on 7/16/2011, 6:44 AM
Then there is Video Enhancer, but you should deinterlace for it as well, or at least deinterleave the fields into two progressive half frames.
Marc S wrote on 7/16/2011, 9:29 AM
I've been using the Yadif deinterlacer plugin for Vegas (free) and it works well. Make sure you set the clip properties to progressive.

http://www.yohng.com/software/yadifvegas.html
farss wrote on 7/16/2011, 3:40 PM
De-interlacing would not be a good idea at all and using fractals could produce some "interesting" outcomes.

This looks more like what you want:
http://www.infognition.com/super_resolution_vdf/

Bob.
PeterDuke wrote on 7/16/2011, 5:37 PM
From Super Resolution web page:

"This plugin shares the Super Resolution engine with Video Enhancer, so in the comparison see the results of Video Enhancer. "

From Video Enhancer FAQ:

"How to upsize interlaced video?
Super-resolution method in Video Enhancer works only with progressive video. If your original video is interlaced (like DV), you've got two options: deinterlace it before SR, or keep proper interlacing.
1. To deinterlace video before upsizing switch to Advanced mode, add one of deinterlacing VD filters (I personally like Deinterlace "Muksun") and then add SR filter. The output video will be progressive.
2. You can keep interlacing by using filter ViewFields before SR and UnViewFields after SR. These filters decompose interlaced frame into two separate pictures (fields) one under another and then combine them back. So each field will be upsized independently."
johnmeyer wrote on 7/16/2011, 6:04 PM
Since we have had all these fun threads about the best way to deinterlace video and also the best way to down-res video from HD to SD in order to create high-quality DVDs from HD sources, perhaps we need another marathon thread about going the other way?

In the meantime, I just did a little testing of my own. I was intrigued by the Super-Resolution plugin, so I downloaded the trial and tested it on some NTSC DV video.

First, I had to figure out how to apply the SuperRes filter to interlaced video. While there are more sophisticated methods, I just followed Neuron's advice in this FAQ:

I want to apply a filter to my interlaced video but the filter does not have an option for interlaced source video

I created a clip at 1440x960 (exactly 2x) because the SuperRes filter is hard-wired to do a 2x scale (it upscales or downscales from that resolution using more standard resolution techniques).

I then rendered using Vegas with "Best" selected, and created a 1440x960 clip.

Finally, I frameserved into three different AVISynth scripts.

I put all the clips (the original and the five up-res'd version) stacked on the Vegas timelines, set the project resolution to 1440x960, made sure all the clips were set to interlaced (I used HuffYUV as the output codec, and it doesn't include an interlaced flag), and looked at the results.

The SuperRes created very nice details in things like people's teeth, and in general seemed to bring out better detail than did the Vegas up-res or the first two simple AVISynth scripts I created.

However, knowing from our previous discussions about resolution that the method of deinterlacing used during the re-sizing makes all the difference (even when the result is still going to be interlaced, you have to at least separate the interlaced footage into separate fields, a process that is done even more precisely when advanced motion-estimation algorithms are used), I then used QTGMC (Nick Hope's favorite tool) and did one more re-size using Spline64Resize.

The quality of this re-size surpassed, in every measure, the result from the commercial SuperRes product. It did an equally good job at "creating" detail that seemingly wasn't there before, but did a much better job handling jaggies.

If one really wanted to get into this, there are all sorts of additional resolution enhancement "tricks" that could be applied, including motion-compensated smart sharpening.

Here's a small section of the original (I cropped using pan/crop), copied from the Vegas "Best" preview:




Here (below) is the SuperRes result. The diagonal lines are their "watermark" (I was using the trial).





And finally here is the result from frameserving into an AVISynth script that deinterlaces using QTGMC, then does an upres using SplineResize64, and finally reinterlaces.




[edit]P.S. The jaggies shown here really are jaggies; they are not interlacing lines. I verified this by serving each clip back out of Vegas into a one-line AVISynth script that separates each frame into its two fields, and the doubles each field to get a full-height frame. This way, I can see the information in each field at one point in time, rather than looking at two fields from two different moments in time, blended into one frame. Doing this allowed me to see that the jaggies really do exist in the Super-Res version, but not in the amazing QTGMC result. Note in particular the jaggies around her had, and on the glasses temple bracket.

The only slight edge I see in the SuperRes version is that the guy's beard may have a very slight bit of additional detail. Hard to say ...




farss wrote on 7/16/2011, 6:55 PM
I would have thought unfolding framesi nstead of actually de-interlacing would preserve as much data as possible before using the SuperRes algorithm. What's probably happening with QTGMC is much the same algorithm is being used.
Where it gets interesting is using images from more than one camera and this is where research is being done now.
The other question is over how many frames is the search being made and how much intelligence is used.
A couple of conclusions I reached some time ago, some of which are mentioned on the SuperRes site that work against good intelligent temporal upscaling:.
Lossy temporal compression.
Noise.
Chroma subsampling.
Detail enhancement.

Good quality Digital Betacam seems to upscale very well. VHS is very problematic to do anything with.

Bob.
johnmeyer wrote on 7/16/2011, 8:50 PM
I would have thought unfolding frames instead of actually de-interlacing would preserve as much data as possible before using the SuperRes algorithm.I too believed that until this spring, when I spent a lot of time understanding what really happens during resolution changes made to interlaced material. Like you, I quickly grasped that you cannot resize the whole frame because odd fields represent a different moment in time from even fields. With fast moving action, almost any function -- but especially resizing -- produces horrifically awful results if you don't first separate the fields.

However, what I didn't realize until I read a lot of posts by people over at doom9.org is that merely separating the fields, then resizing, and then recombining the fields can also produce fairly awful results unless you are resizing by exact multiples of two. The reason is that the new fields need to be in spatial positions that are part way between the original fields.

You can see why this is so if you draw four horizontal lines, one beneath the next, equally spaced apart. Thus, you have a line at the top of the page, one at the bottom, and two at the one-third and two-third position on the page. Then, starting with the same top and bottom lines, now draw four equally spaced lines top to bottom so that you have a total of six lines in this new drawing instead of four.

You have now re-sized from a resolution of 4 to a resolution of 6.

If you did this correctly, you will see that none of the new lines (except the ones at the top and bottom) are in the same position as the original lines. In order to create these new lines (if they are actually representing video fields), you need to somehow get information from the original four lines, none of which (except the top and bottom lines) are in the same points in space as the original lines. So, the only way to do this accurately is to do an exceptionally smart job of deinterlacing so that you end up with twice the number of frames, with the fields in each frame being a smart estimation of what those frames would look like if the video had been taken as progressive video. Once you have these estimated fields coming from the same moment in time (unlike the original odd and even fields which came from different moments in time), you can rather easily create the intermediate fields by averaging adjacent fields together.

Once you do that, then you can "unfold" to get back to the original frame rate, and to get back to real interlaced video.

If you don't do things this way, and simply "unfold" and then "refold" fields before and after resizing, you'll end up with many of the same ugly artifacts that you get if you try resizing the interlaced video without doing any field separation at all. The artifacts may not be as strong, but they'll still be very obvious.

As can be seen in my still shots above, the differences are not at all subtle. What's more, the differences when you actually view the video in motion (which is the real test), the differences can be even more striking.

The key to everything is having world-class motion estimation. As I've posted here many times before, I've tried for six years, via PMs to the development team, to get them to invest in this technology. With it, you can do world-class noise reduction, scaling, slow motion, bad frame replacement, and countless other tricks. The company that can master this technology can offer features that no other editing or effects program can match. AE clearly has invested in this, and perhaps it doesn't belong in an editor, but since scaling is such an essential part of so many editing workflows, and since scaling without using this technology is clearly inferior, I argue that Sony should invest in the technology and should include it in Vegas.

But what do I know ...

[edit]P.S. You are absolutely correct in your statements about noise reduction. Almost anything done with video -- scaling, encoding, masking, etc. -- is dramatically improved if you can first get the video as clean as possible, free from all extraneous artifacts.
NickHope wrote on 7/16/2011, 9:18 PM
John, I'd be very interested in a Spline64Resize vs Lanczos4Resize comparison for upscaling, if you're in the mood. There is also BlackmanResize, that this QTGMC-based helper script uses for upscaling.
musicvid10 wrote on 7/16/2011, 9:29 PM
"John, I'd be very interested in a Spline64Resize vs Lanczos4Resize comparison for upscaling,"
I see yet another megathread on the horizon . . .
johnmeyer wrote on 7/16/2011, 11:34 PM
John, I'd be very interested in a Spline64Resize vs Lanczos4Resize comparison for upscaling, if you're in the mood. There is also BlackmanResize, that this QTGMC-based helper script uses for upscalingI'm not sure there is much to be gained going down that road. I've read (as I'm sure you have as well) dozens of threads over in doom9.org about the advantages of one resizer vs. another. From what I've read, some do better with motion, some better with stills; some have issues with various chroma situations, and others don't; and so on.

However, I don't think the differences are all that great, compared to the differences I illustrated that can be gained by doing correct estimation and interpolation prior to the resizing. Much of this has already been covered in the earlier threads on downsizing, and not much is going to be different going in the other direction.

But, when going the other direction (upres'ng) there is the additional burden of trying to estimate detail that wasn't in the original. The "SuperRes" product mentioned in the initial posts supposedly does this by collecting information from adjacent frames and then cleverly averaging them together. Most of the experts over at doom9.org "pooh-pooh'd" that technology and claimed it can't work. I have no opinion one way or the other, and haven't tested it enough to be able to tell for sure whether it doesn't do any better than what I posted with QTGMC.

If anyone really wants to do additional testing, I think most of the additional quality gains will be found by doing inventive things with adaptive sharpening. A good example of what can be done is shown in this screen grab I posted in doom9 about a year ago:



The image on the left is the original resolution from an 8mm film frame. The image on the right is what I was able to create using techniques created by "VideoFred" using multiple levels of smart sharpening. I can guarantee that in none of the adjacent frames do any of those vertical porch posts show up as cleanly.

Of course we can argue as to whether I have created detail that wasn't there, and I probably have not. However, the perception to the viewer is that the image on the right is not just sharper, but does indeed contain more detail.

So, once you do the first step using the techniques I posted above, and once you do some denoising, as Bob (farss) suggests, I think that adaptive sharpening is where most of the research should go.

Of course, if you have the patience to search through the doom9 posts, someone over there has probably already covered this territory and has some pretty clever solutions.
farss wrote on 7/17/2011, 12:40 AM
I just found this interesting paper:
Image Super-Resolution as Sparse Representation of Raw Image Patches

I will not even pretend to understand anything more than the concept and even then I'm not certain I fully understand it. The fact that it uses dictionaries is most interesting though, if I understand the concept correctly this relies on understand what the patches of the image should look like.


Bob.
PeterDuke wrote on 7/17/2011, 3:30 AM
That idea may work for something as specific as say faces, but you would need a very large dictionary to hold models of all the elemental shapes in the world.
PeterDuke wrote on 7/17/2011, 3:52 AM
johnmeyer has already pretty well killed the idea of working with pairs of noninterlaced fields rather than "proper" deinterlacing, but if the advantage of Super Resolution/Video Enhancer over other methods is that it can utilize adjacent frames to get better interpolation, then pairs of noninterlaced fields is not the best way to go.

In this method, each field would be augmented by every second adjacent field, not all and the closest fields.

On the other hand if each field were first converted to a full frame (double rate deinterlacing) then adjacent fields could be used in the interpolation. Adjacent fields would be best to use, of course, because they would have least change from the field under consideration, compared to other fields.

I personally believe that using adjacent frames would be beneficial if the scene is fairly static (elements move very slowly) and the camera has slight movement as in hand held, or, if the camera is steady then the scene as a whole moves past. slowly. These constraints could be relaxed if we used motion compensation on the elements... Now we are delving into techniques for slo-mo and what works best for it! Denoising came up there too.