Strange Quality issues on rendered playback

PeteS160 wrote on 4/29/2024, 9:31 PM

I am a long time user but this is the first time I've reached the point where I need to ask for help because I'm at out of idea's and can't seem to figure out what's going on.

I'll start with PC/Software specs...
Vega's Pro 21, Movie Studio Platinum 17, Vegas Movie Studio 16 Suit(Platinum), Vegas Movie Studio 13(Platinum), Vegas Movie Studio 11 HD. I'm by no means new to Vegas and have been using it since around 2010.

PC 1: AMD 5800X3d, 64gb ram, AMD 6900 XT(Black Edition), system is fully liquid cooled and has multiple M.2 drives(tons of very fast storage), Windows 10 fully updated, AMD Content Creator drivers(tried multiple versions)

PC 2: AMD 3800X, 64gb ram, AMD 5700 XT(Special Edition), multiple M.2 drives(tons of very fast storage), Windows 10 fully updated, Standard AMD drivers(have tried multiple versions)

So long story short, I created a bunch of short video clips in After Effects that I used to build my project in Vegas. I rendered the clips in After Effects at 1920x1080 60 FPS and they are in an uncompressed(lossless) AVI format. Each one of the clips is playable with both Windows media player or VLC and will play back silky smooth despite the insanely high bitrate. The preview in Vegas will also play the clips back in full screen preview...not quit as nice looking as they play in VLC but the preview is still very smooth and high quality.

Now the issue I've run into is with rendering my final project.....I've rendered hundreds of videos with various versions of Vegas over the years using all sorts of input media going back to the days of Fraps and Dxtory where you would have to piece gave footage back together from dozens of 4 gig sections(in avi format) and even back then never ran into an issue like what I'm dealing with now.

So, my issue is that when the project is rendered using MAGIX AVC/AAC as H.264/X.264 format it will loose a MASSIVE amount of visual quality unless I'm using stupid high bit rates(120,000+kbps average) and even then there are parts where the video will just look REALLY bad. The MAGIX HEVC/ACC has similar issues, while it can get away with a lower bitrate the overall visual quality I would say is much....much worse looking then the AVC format. Now where this gets interesting is that the Sony XAVC S Long looks SIGNIFANCTLY better however it causes a frame rate mismatch(59.94 VS 60) and it seems it takes a pretty decently powered PC to be able to play back the video(using the Sony XAVC format) with out stuttering/chopping or just locking up. The Sony format produces a massive file...around 32 gigs for a little over 20 mins and the bitrate is still fairly high. The Magix AVC WILL play back on a TV(over USB) but it looks pretty bad. The best way I can describe it is that it looks like its lagging and will start dropping frames trying to catch back up. I've tested this on multiple tv's and at both 1080/4K and they all have the same general issues.

Up until this point I had been working in Movie Studio Plat 17 and decided to upgrade to Vegas Pro 21 just to get access to more options for rendering with and was hoping that a newer version may have improved the way it was rendering the video but that didn't turn out to be the case. At that point I decided to give Voukoder(Classic) a try and I saw MASSIVE improvements all the way around. The video quality is now almost if not the same quality as the source footage I used to build my project with, the file size is a fraction of what I was seeing with any of the Magix formats or with the Sony codec. The bit rate has come down so much that I was almost in disbelief when I played back the first clip....like I was actually shocked at how much the quality improved.

Now, while the picture quality of the video is almost flawless and it will play back silky smooth in VLC it will struggle in Windows Media play/Windows Movie play or really any other type of player other then VLC. It also struggles on YouTube and looks like it's still dropping frames....for some reason this video also takes an ungodly long time for youtube to process. It's been taking around 1 1/2 days for them to butcher it....I mean process it. Just for comparison I uploaded another video made with video game footage that was rendered using the same settings and had a similar file size that only took around 2 hours to get processed into 4K....so there's something else going on with this footage from After Effects that Youtube really doesn't seem to like. The video still does not playback on a TV smoothly either despite the improvements Voukoder made to the bitrate.

The best playback and quality I've gotten so far was with the details below.
Voukoder HEVC(X265)
File Size: 552mb
Length: 24:18
Data Rate: 2,852 kbps
Total Bitrate: 3,167 kbps

crf=19.000 limit-sao=1 no-open-gop=1 preset=slow x265-params=allow-non-conformance=1:aq-mode=4:aq-motion=1:qg-size=16:crqpoffs=-2:rd=3:ctu=64:min-cu-size=16:limit-modes=1:rect=1:early-skip=1:rskip=1:splitrd-skip=1:fast-intra=1:tskip-fast=1:rdoq-level=1:tu-intra-depth=2:tu-inter-depth=2:limit-tu=1:rdpenalty=1:max-merge=2:me=umh:subme=5:merange=48:weightb=1:no-mcstf=1:min-keyint=5:no-hist-scenecut=1:radl=2:b-adapt=2:bframes=11:ref=3:fades=1:sao-non-deblock=1:deblock=0:hash=2

Using these settings the video will play back in VLC and look PEFECT. In Windows Media player overall it looks and plays Ok to someone with average or less then great eye sight, my Wife thinks it looked fine until I played it side by side in VLC and then she was able to notice the difference. In media player the best way I can describe it would be that it's like the frame rate is going up and down where some places its the full 60 FPS and other places its dropping to like maybe 30 or 40 FPS. Now I know the frame rate doesn't change, Media Info reports that its frame rate mode is constant and its 60fps so I have no idea why Windows media player....Youtube and every TV I own all seem to have the same type of issue when trying to play this back.

At this point I'm open to suggestions because I'm honestly out of idea's. I feel like there must be a setting in Voukoder that needs to be turned ON/OFF or have the value changed but I have no idea what even half of the advanced options are for and I haven't been able to find much information on what most of the stuff even does.

Comments

RogerS wrote on 4/29/2024, 9:55 PM

Really high peak bitrates may be unplayable on TVs or devices.

Anyway it might save time if you could share a screenshot of your project properties and render setting and a link to download a sample clip of where you see quality issues (or bad vs good). We can see if it's an issue with the file or the playback device.

mark-y wrote on 4/29/2024, 10:13 PM

So long story short, I created a bunch of short video clips in After Effects that I used to build my project in Vegas. I rendered the clips in After Effects at 1920x1080 60 FPS and they are in an uncompressed(lossless) AVI format. Each one of the clips is playable with both Windows media player or VLC and will play back silky smooth despite the insanely high bitrate. The preview in Vegas will also play the clips back in full screen preview...not quit as nice looking as they play in VLC but the preview is still very smooth and high quality.

Provide one of your source AVI clips by uploading it to Drive or Dropbox (not the forum or social media) and post the link here. Using a highly compressed codec (HEVC) may be counterintuitive since you seem to be stating you favor quality over small file size.

 

PeteS160 wrote on 4/29/2024, 10:57 PM

So long story short, I created a bunch of short video clips in After Effects that I used to build my project in Vegas. I rendered the clips in After Effects at 1920x1080 60 FPS and they are in an uncompressed(lossless) AVI format. Each one of the clips is playable with both Windows media player or VLC and will play back silky smooth despite the insanely high bitrate. The preview in Vegas will also play the clips back in full screen preview...not quit as nice looking as they play in VLC but the preview is still very smooth and high quality.

Provide one of your source AVI clips by uploading it to Drive or Dropbox (not the forum or social media) and post the link here. Using a highly compressed codec (HEVC) may be counterintuitive since you seem to be stating you favor quality over small file size.

 

I can do that however it'll take me a day or so to upload load it. The smallest section from After Effects I want to say is around 25 gigs for a 1 minute section.

 

Format                       : AVI
Format/Info                : Audio Video Interleave
Format profile            : OpenDML
Format settings         : BitmapInfoHeader
File size                    : 25.7 GiB
Duration                     : 1 min 13 s
Overall bit rate         : 2 986 Mb/s
Frame rate               : 60.000 FPS
Recorded date          : 2024-04-17 11:36:42.452956

Video
ID                   : 0
Format           : RGB
Codec ID       : 0x00000000
Codec ID/Info:Basic Windows bitmap format. 1, 4 and 8 bpp versions are palettised. 16, 24 and 32bpp contain raw RGB samples
Duration        : 1 min 13 s
Bit rate          : 2,986 Mb/s
Width            : 1920 pixels
Height           : 1080 pixels
Display aspect ratio   : 16:9
Frame rate                 : 60.000 FPS
Color space               : RGB
Bit depth                    : 8 bits
Bits/(Pixel*Frame)     : 24.000
Time code of first frame : 00:00:00:00 / 00:00:00:00
Time code source            : Adobe tc_A / Adobe tc_O
Stream size                    : 25.7 GiB (100%)

john_dennis wrote on 4/29/2024, 11:07 PM

@PeteS160

Download and try this AVC Voukoder setting and report back.*

CRF22, GOP30, GOPMin15, No B-Frames.sft2

Location to place the file:

C:\Users\username\AppData\Roaming\VEGAS\Render Templates\voukoder

*You may have to unblock the file after downloading to get it to work.

mark-y wrote on 4/30/2024, 12:11 AM

👍

PeteS160 wrote on 4/30/2024, 12:27 AM

@PeteS160

Download and try this AVC Voukoder setting and report back.*

CRF22, GOP30, GOPMin15, No B-Frames.sft2

Location to place the file:

C:\Users\username\AppData\Roaming\VEGAS\Render Templates\voukoder

*You may have to unblock the file after downloading to get it to work.

While the speed of that preset is very impressive, the image quality leaves a good bit to be desired still although it didn't seem to have any of the micro stuttering in Windows media player.

I used your preset on one of the raw sections and the resulting video was 35mb with a bit rate of 4,657. Just for comparison I rendered the same clip using the Voukoder HEVC X.265(CPU) preset I've been working with and that file was 16mb with a bit rate of 2,091. The images look much better in mine.... however as I expect there are 2 small stutters that I noticed when it was played in Windows Media player.

Just for comparison purposes I ran the version I rendered using your preset through NMKODER and the VMAF score was 91.414432 so its not just my imagination that your preset reduced the image quality by a noticeable amount.



 

john_dennis wrote on 4/30/2024, 12:53 AM

@PeteS160 said: "... its not just my imagination that your preset reduced the image quality by a noticeable amount."

To regain some image quality, lower the CRF number incrementally to 20. Leave the GOP alone.

I've just created my own GOP Rule of Thumb. "GOP should never be greater than one's age or the frame rate, whichever is lower."

I don't have any use for HEVC.

mark-y wrote on 4/30/2024, 1:46 AM

Pete, you can render a few seconds of uncompressed AVI from your source without changing a thing. Since you share impressions of lower quality with skeletal AVC, that still seems to be the next logical step.

RogerS wrote on 4/30/2024, 1:50 AM

Download and try this AVC Voukoder setting and report back.*

CRF22, GOP30, GOPMin15, No B-Frames.sft2

Location to place the file:

C:\Users\username\AppData\Roaming\VEGAS\Render Templates\voukoder

*You may have to unblock the file after downloading to get it to work.

While the speed of that preset is very impressive, the image quality leaves a good bit to be desired still although it didn't seem to have any of the micro stuttering in Windows media player.

I used your preset on one of the raw sections and the resulting video was 35mb with a bit rate of 4,657. Just for comparison I rendered the same clip using the Voukoder HEVC X.265(CPU) preset I've been working with and that file was 16mb with a bit rate of 2,091. The images look much better in mine.... however as I expect there are 2 small stutters that I noticed when it was played in Windows Media player.

Just for comparison purposes I ran the version I rendered using your preset through NMKODER and the VMAF score was 91.414432 so its not just my imagination that your preset reduced the image quality by a noticeable amount.
 

You can't really compare bitrates between AVC and HEVC (well, if you choose to, 2 to 1 could be reasonable for a similar quality file).

Regarding stuttering you are sure that all framerates match between source, project and render?

If so, the limited peak bitrate may be what's helping the Media Player and TV. Or lack of b-frames or smaller GOP. which makes the file easier to decode. Try testing combinations of the 3 to get acceptable quality in a playable file.

For VMAF what are you targeting for quality- 95 or so?