(FAQ) How can I improve the quality of my AVC (H.264) renders?

NickHope wrote on 11/25/2016, 12:53 AM

IMPORTANT! VEGAS Pro 15 introduced the following changes:

A new MAGIX AVC/AAC codec significantly accelerates encoding on machines that utilize Intel® Quick Sync Video (QSV) technology
A new MAGIX AVC/AAC codec significantly accelerates encoding on machines equipped with modern NVIDIA graphics card GPU technology

Effectively the MainConcept AVC/AAC encoder has been replaced with the MAGIX AVC/AAC encoder, which inherits the same functionality and adds rendering acceleration via QSV or NVIDIA NVENC (supported NVIDIA GPUs) and, as of build 361, AMD VCE.

This FAQ post will be updated to reflect these changes when their effects are better understood. In the meantime, it remains out of date and applies to VP14 and earlier.

Discussions:

-----------

ORIGINAL POST FOR VEGAS PRO 14 AND EARLIER...

AVC and H.264 are the same thing. It is a modern video format with wide support and high-efficiency compression, meaning that (if done properly) it gives high quality per file size. It is a very popular format for both offline playback, web playback, and upload to video-serving platforms such as YouTube and Vimeo. This post concentrates on those destinations.

YouTube etc. will re-render your uploaded file and it will lose quality. This article is about giving them the best chance of making a good job of it.

VEGAS Pro has 2 built-in encoders: Sony AVC/MVC & MainConcept AVC/AAC. They are generally similar and generate video/audio of the same format.

1. PREPARATION

Firstly your media should look and behave correctly on your timeline.

If your video won't open properly in VEGAS, or looks corrupted, read this: Why won't VEGAS open my file correctly? Codecs, Converters etc.
You should then make sure to match your project properties, set the resample mode etc. before rendering. DON'T SKIP THIS!
You should also be aware that your rendered video may play back on the web or in a local player with more contrast than you see in the VEGAS Pro preview window and take steps to rectify that if necessary.

Let's first assume that your source video is progressive (not interlaced), as this format is becoming more and more common and is easier to deal with (you can use MediaInfo to see if it's progressive or interlaced).

This post assumes you wish to render 8-bit, standard dynamic range (SDR) video. Not 10-bit, HDR etc..

YouTube's officially recommended upload encoding settings are here. Vimeo's are here. The bit rates are are rather low if maximum quality is your priority. Also see this social media cheat sheet for video specs.

2. SONY AVC/AAC ENCODER

This is the faster of the 2 encoders (unless you can use legacy GPU rendering).

The maximum resolution 1920 x 1080
The maximum frame rate is 59.940fps
The maximum bit rate is 25,999,360bps

To start, open one of the 2 "Internet..." templates from "Render As" > "Output Format" > "Sony AVC/MVC" that nearest matches your project/target resolution and change the frame rate to match your project properties (which should usually be the same as your source media unless it is more than 59.94fps).

To improve the quality, increase the "Bit rate (bps)" setting. Even though a value does not appear in the drop-down menu and there is no "custom bit rate" option, you can over-type it in manually in the white field. For a 1920x1080-30p video you could try increasing the bitrate from 16,000,000 to 20,000,000bps or higher. If you are uploading a high frame rate (48/50/59.940p) 1920x1080 file then you would probably want to use the maximum available bit rate of 25,999,360bps:

3. MAINCONCEPT AVC/MVC ENCODER

This is generally the better quality encoder. It is slower than the Sony AVC encoder. You can speed it up a lot with legacy GPU rendering, if you have an old graphics card such as an NVIDIA GeForce GTX 580 or AMD Radeon HD 6970, but the quality will suffer a little.

The maximum resolution is 4096 x 2304*
The maximum frame rate is 172fps*
The maximum bit rate is 240,000,000bps

* Extreme combinations of high resolution & frame rate cause either "invalid argument" or low-memory failures but I was able to render 1920x1080@172fps and 4096x2160@59.94fps.

To start, open the template from "Render As" > "Output Format" > "MainConcept AVC/AAC" that nearest matches your target settings (e.g. "Internet HD 1080p") and change the frame rate to match your project properties (which should usually be the same as your source media unless it is more than 59.94fps). Even though a high frame rate value (e.g. 48, 50, 59.94) does not appear in the drop-down menu and there is no "custom frame rate" option, you can over-type it in manually in the white field.

To improve the quality, check the "Two-pass" box, the "Use deblocking filter" box, and if necessary increase the maximum and average bit rates. Again, you can type in custom values. The appropriate bit rates depend on the movement and detail in your video, as well as the resolution and frame rate, so it is difficult to make specific recommendations. For a 1920x1080-30p video you could try increasing the Maximum (bps) to 28,000,000 and the Average (bps) to 16,000,000. For a high frame rate (48/50/59.940p) or UHD (3840x2160) file you will probably want higher, for example:

4. x264 EXTERNAL ENCODER

x264 is a free, open-source codec that enables you to achieve higher quality for the same file size as the built-in VEGAS AVC encoders. It may also be faster than the VEGAS encoders, especially without GPU rendering.

Although you can render with x264 from a command line, there are many free apps that provide a GUI to use the x264 codec to render AVC. These include Handbrake (x64), Staxrip (x64), MeGUI (x86/x64), RipBot264 (x86) etc..

4a. Render Intermediate

You can render a high-quality intermediate file from VEGAS Pro and open it in a rendering app for transcoding to AVC. Be sure to choose or save a template that matches your project settings. Popular formats for intermediates include XAVC Intra (included in VEGAS, but the rendering app might not read it), Cineform (installed with GoPro Quik Studio), MagicYUV, DNxHD etc..

4b. Frameserving

If the rendering app is 32-bit (x86) (e.g. MeGUI) then you can frameserve to it using the Debugmode Frameserver instead of rendering an intermediate. This saves you time and storage space because it writes the intermediate file from VEGAS only as it is required by the encoder.

After installing Debugmode Frameserver, it appears as an Output Format in the VEGAS Pro Render As window. If you have Frameserver already installed for Vegas Pro 13 you can get it working in VEGAS Pro 14 by copying:

C:\Program Files\Sony\Vegas Pro 13.0\Frameserver.x64.fio2007-config
to
C:\Program Files\VEGAS\VEGAS Pro 14.0\Frameserver.x64.fio2007-config

The x86 apps, MeGUI and RipBot264 will accept files directly from Frameserver. x64 apps will not. The x86 version of MeGUI has not been developed since 2015 but its plugins including x264 can be manually updated.

Another option is to install 32-bit AviSynth and 32-bit ffmpeg (help) and to use a Windows script to render x264.

Note that there are 2 bugs with Frameserver in Vegas Pro 12 and later. Firstly it will only serve audio at 44.1kHz. That is not a major problem for web video, as much of it is delivered at 44.1kHz, although YouTube are now recommending a sample rate of 48 or 96kbps. Secondly, the last second of the audio stream is corrupt and extends beyond the video stream. You can ignore this if the very last part of your video is silent or unimportant, or you could work around it by rendering audio separately from Vegas at 48Khz and remuxing with the video, or you could use the following process...

4c. Happy Otter Scripts

The RenderPlus script within Happy Otter Scripts provides a convenient interface for rendering from VEGAS Pro using the x264 codec (and other codecs) via Debugmode Frameserver. The required installation and configuration of the third-party tools such as Frameserver and x264 is handled by the Happy Otter Scripts installer, and it compensates for the Frameserver bugs mentioned in part 4b.

For a straightforward render for upload to YouTube etc. it is recommended to use the "Simple" Render Mode and to choose either the x264-good quality-medium-crf 23-aac192 or x264-high quality-slower-crf 18-aac 320 encoder template. I usually use the latter, but note that it is slower and the file size is larger than the "medium" template.

It is also possible to access all sorts of AviSynth processing and encoder settings using the Advanced render mode, including GPU rendering, slow motion, denoising etc., or to do an x264 render using the AviDub script within Happy Otter Scripts.

4d. Vegas2Handbrake

[Edit: See this article for updated instructions for using Vegas2Handbrake. HappyOtterScripts (above in part 4c) is generally now a superior method]

Vegas2Handbrake (see here if that link doesn't work) automates the process of rendering with Handbrake and is popular with many community members for rendering AVC. By carefully following the instructions to add a 1 second buffer you can also compensate for the Frameserver audio bugs.

Instructions are included in the download. If you are using VEGAS Pro 14 then you need to make these changes to get Vegas2Handbrake working.

Since Handbrake 1.0.1 was released, you also need to make the changes described in this post. I summarize the situation in this comment.

4e. x264 Settings

After much discussion and testing of numerous options, my recommendation for general use is to leave all the x264 options at their default settings and adjust the quality/file size simply by adjusting the Constant Rate Factor (CRF). Note that (suprisingly) a lower crf value = higher quality and larger file size. Suggested values are 20.0 to 23.0. For critical YouTube uploads I have often gone as low as 18.0. The CRF value will usually be presented as a "Constant Quality" or just "Quality" figure in the app's settings. In Handbrake it looks like this:

Given a choice, render to an MP4 container rather than MKV. Also check the "Web Optimized" setting if you have one.

5. INTERLACED FOOTAGE

If your source footage is interlaced, you should deinterlace it to make it progressive for web upload. In a video with significant motion, "Interpolate fields" (in "Project Properties" > "Deinterlace method") does a poor job and "Blend fields" a worse one. Here are some higher quality options:

5a. VP14 smart adaptive deinterlacing

VEGAS Pro 14.0 has a new smart adaptive deinterlacing feature that does a great job. Set "Project Properties" > "Field Order" > "None (progressive scan)" and "Deinterlace method" > "Smart Adaptive (GPU only)", then render using progressive settings.

5b. Handbrake Decomb

If you are using Handbrake then leave your project properties as interlaced ("Field Order" = "Upper Field First" or "Lower Field First" to match your footage), frameserve (or render an intermediate file) in interlaced format, and check "Decomb" on Handbrake's "Filters" tab for a high quality result.

5c. Yadif Deinterlace for Sony Vegas

The Yadif Deinterlace plugin for Sony Vegas does not give as good a result as the other 3 options in this section but it is much better than "interpolate" or "blend" and is simple to use. It doesn't only work in Vegas Pro 10. It works in VP11/12/13/14 too. Install it in one of these example locations, depending on your system:

C:\Program Files\Common Files\OFX\Plugins
C:\Program Files\VEGAS\VEGAS Pro 14.0\OFX Video Plug-Ins
C:\Program Files\Sony\Vegas Pro 13.0\OFX Video Plug-Ins

Set you project and render settings to progressive scan and apply the plugin as a video FX to the event(s) or track or video output. Here are the settings I use for my HDV footage:

5d. QTGMC in AviSynth

For maximum quality, export to AviSynth and use the QTGMC deinterlacing script, then render using x264 in, for example, MeGUI. This is complicated (Google is your friend). The previous 3 options described are not far behind in quality and much simpler to set up. Update: The Happy Otter Scripts RenderPlus script described in part 4c above, provides straightforward access to QTGMC deinterlacing in its advanced mode.

6. RELATED POSTS