Hi Bob, how've you been and how's the summer in Australia? Summer here in Brazil is being great. Well, let's get back to the subject...
"I think we need to know specifically what you mean by aliasing"
- the effect itself - the pics comming from the center of the screen, flying...their edges look like saws (hope you know what I mean).
" I do know that TrueSpace has an AA and Motion Blur filters..."
- True Space?
- One more thing, I'm seeing the problem on a regular tv set.

It's still not clear to me what you mean by antialiasing, but allow me to give an example of what happened to me and how I corrected it:

When I make a DVD I usually make motion menus, which means the screen is filled with smaller windows that have video playing in them from the various chapters represented. The problem is that if you take a 10-second clip from that chapter and shrink it down using Track Motion in Vegas, all sorts of strange artifacts appear. This is because there is too much spatial information in that frame than can be represented by the limited number of pixels.

The solution is simple--just blur the video before doing the shrinking. You'll have to experiment to see how much blurring is necessary. Too little and you get aliasing; too much and your small video is blurry.

In one menu I did last year, the motion menu videos began as full-frame videos, which then began to shrink one-by-one as they moved into their final positions in the menu structure. For that effect, I had to keyframe each video so that it started out at full resolution and then became more blurry as it shrank. In such a manner I was able to get a very nice shrinking effect without any aliasing effects showing up as the image shrank.

It's really very easy to do, though it takes some concentration. I can't have a glass of wine in the evening while I'm doing it, or I'll screw something up.

Thanks for responding, riredale, but it's got nothing to do with blur... Just think about a square, a simple one, turning right in the center of the screen....I'm talking about its edges.....sharp edges that look like a saw, got it? Antialiasing is something people involved in 3D know a lot. Because of the pixels, a rounded edge may look sometimes as a saw and not that round. Antialiasing filters or high levels of antialiasing only relieve the problem, making the edges look more softened. That's it. So, what do you think?

I think the problem is Vegas isn't a 3D app. Therefore it doesn't understand objects as such nor is it rendering from vectors to bitmaps. So when you take your square and rotate it there's no way for Vegas to apply AA even if it supported it. 3D applications like Maya, Lightwave and TrueSpace are converting vectors into bitmaps, same as video game rendering so they or the video GPU knows how to apply AA. For example if you have a vertical line made up of a column of pixels and rotate that 45deg Vegas can only remap all the pixels, If a line is defined by two points then the software knows about the angle and how to apply AA by filling in adjoining pixels (sub pixel rendering) ti give an apparently smooth edge. Now probably you could fudge this is Vegas by adding blur etc but you'd have to do this on a case by case basis which would get very tedious. That's why it's far better to do these sorts of things in 3D applications that are vector based.
You can see this in PS as well as Vegas too. If you rasterise something (say text) in PS and then rotate it you can get bad aliasing. If you rotate the text before rasterising then the AA filtering can work and you get a smooth result.

Something you could try in Vegas which may or may not help is to upscale the video first to say 1080 and do your thing there and then render down to SD. This isn't as bad as it sounds as you don't need to render to HD, just use the footage in a HD project, Vegas I believe upscale to the higher res, applies the FXs or motion at that res and then downscales for the render out. It will sure slow your renders down something terrible but will not eat up disk space. Try it on say 10 secs of video first to see if it helps. You can add very small amounts of things like Guassian Blur (.001) in your HD project and see if that helps.

Summer down here has been pretty hot and dry, water restrictions are getting worse but at least no bushfires so far this year.

Bob.

Nice to hearing from you farss.
"That's why it's far better to do these sorts of things in 3D applications that are vector based."
- I disagree, the effect - those small screens that come out of the center of the screen and fly away, were in a Veggie file. When you render the whole thing, the edges of those small screeens, let's say at 45º look like this (like a saw):
/|/|/|/|/|/|/|/|/|/|/|/|
I'm sorry, but I guess Photoshop can help with its AA filter - can't it????? (haven't tried it yet)

"Something you could try in Vegas which may or may not help is to upscale the video first to say 1080 and do your thing there and then render down to SD. "
- This method is also used with Lightwave 3D. When you don't want to apply AA, you render the way you've just mentioned. As you render the whole thing down, AA will be there ;)

"Summer down here has been pretty hot and dry..."
- 33º C in São Paulo (located in the countryside); 35º~40ºC on the beach ;) So what about going surfing in Ubatuba?

PS: Since we cannot upload images around here, I may add a new thread on this subject at www.videohelp.com

C'mon guys! It's nice to hearing from you all...I'd like to get some new ideas on how to fix this...Thanks in advance ;)

"This method is also used with Lightwave 3D. When you don't want to apply AA, you render the way you've just mentioned. As you render the whole thing down, AA will be there ;)"
To chime in on this one, Lightwave multi-samples with shifts (much like n-tap filters in 3D accelerators) for anti-aliasing, but features a few clever optimizations (such as adaptive sampling) to keep render times somewhere below stratospheric. Soft filtering in Lightwave, on the other hand, is just a blur. If you spend a lot of time in Lightwave, you'll find things that can cause texture-shimmers and jagged edges even at extreme AA levels, but they are admittedly rare. If we could take as long as Lightwave to render individual frames, we might be able to take more extreme measures.

There are a couple of things that you could mean by a rotating square. The square could be part of a given clip (say, only in the middle third of the clip) with the entire clip being rotated, or the square could be a solid color generator or solid piece of media being rotated in Vegas. In both cases we anti-alias the output. We use different algorithms for different output quality, so I urge you to try Good and Best qualities to see if they provide better results for you. Also keep in mind the aliasing-like effects of interlaced video on a progressive-scan monitor when doing tests. Rendered files can look really poor on your computer while looking perfect on your intended delivery medium (say, television).

Before getting into the meat of the problem, I'll suggest a couple of fixes. Add a Gaussian blur to the track of your rotating square, set it as a post-composite filter, and set it to 0.002, 0.002, Alpha. When you blur only alpha, you end up softening only the edges of your footage and encroaching very little to achieve soft edges. This only softens the edges generated by Track Motion, and it is pretty effective. Another thing to keep in mind is Gamma. If you made a white clip with a red square in it using generators, rendered it to DV, brought it back into Vegas, and rotated it, the edges of the red square might look jagged on your computer monitor or most delivery mediums. To correct this, apply a light gamma correction (using levels). 2.2 is probably the upper limit, and you can apply less aggressive gamma to suit your needs. This trick will also clean up the appearance of outer-edges of footage generated by track motion. If you pseudo-linearize your camera-based footage (by applying a gamma less than 1) and then apply its inverse on your output (say, 0.65 on your footage and 1.538 on your output, all of your edges from motion may look somewhat less edgy.

This brings us to the heart of the problem, which is that anti-aliasing techniques involving sampled data modify or destroy that data in the process. A core problem here is that the highest frequency that can be represented accurately by your sampled data is only half of the sampling rate (go google Nyquist Aliasing for more info), and any measure taken to completely eliminate aliasing when resampling sampled data would eliminate a great deal of high-frequency information. The generally accepted best (for the data) approach is to use what is called a sinc resample. In the frequency domain (and you may be tuning out now, which is okay), a sinc sampling kernel looks like a cliff, and is often called a brick wall. Unfortunately, accurate representation of this sampling kernel requires a very large sampling window (in theory, infinite) to achieve decent results. At least at this time, the time required to render using this method in Vegas would likely make it more of a bullet-point feature than a real feature.

Trust us, we care a great deal about the quality of your output, as it's often ours as well, and Vegas definitely anti-aliases your motion-graphics output.

Did you try copy/paste a screenshot of your work in Photoshop?

Vegas 5 indeed to have a big problem with antialiasing on NTSC preview. But if your render a frame (or a sequence) in bitmap (or video), making shure rendering settings are at best/full, you'l get exactly the same antialiased results that you get from a graphic app like Fireworks for example (or in a vectorial tool like Freehand when you rasterize).

Maybe you could have greater antialiasing control in Photoshop?

Chaboud,
thanks so much for all that info, it's certainly given me a lot of food for thought.
As you may have seen from some of my posts from a while ago I've done a fair amount of work trying to solve the Nyquist problem with hi res stills and got pretty decent results out of Vegas by using a few tricks I'd worked out.
I for one would put up with the long render times that a brick wall filter would involve, my current technique is extremely slow to render anyway. This would have application not only for stills but also when downsampling HiDef video. I'm pretty certain that most of the hardware downscalers use some pretty fancy filtering but even then I seeing more and more aliasing problems in broadcast down here as more material is originated as HiDef.
Can I also suggest that the sort of information you've supplied here (and thanks again) either needs to be in the documentation or in a white paper.
Bob.