Struggling with Pixel Definition in Extremely Dark Scenes

Rivas wrote on 9/30/2025, 6:38 AM

Hello everyone,

I’m fairly new to the world of video editing and rendering, and I’ve been running into a consistent issue that I would love to hear more experienced opinions on.

I’ve been testing Vegas Pro 20’s native renderer as well as Voukoder with my system:

CPU: AMD Ryzen 7 5700X

GPU: NVIDIA RTX 4070

RAM: 32GB DDR4

The problem arises specifically in extremely dark gameplay scenes (for example, night sequences in Dying Light). No matter which settings I try, I always notice heavy pixelation or blockiness in the shadows compared to the original recording.

Here’s what I’ve tried so far with Voukoder:

VBR with very high ranges (up to 150–200 Mbps)

CQP values as low as 15 and even 10

CBR up to 100 Mbps

Unfortunately, none of these configurations look anywhere near as clean as the source file. The only time the problem completely disappears is when I use Lossless mode, but that pushes the bitrate up to 320 Mbps or higher, resulting in massive file sizes (multiple GBs per minute).

As I said, I’m still new to video encoding, so I’d really appreciate insight from those with more experience. From my perspective, I understand there’s probably no “magic trick” here – if I want an absolutely flawless result in these extremely dark scenarios, I need to work with lossless files. Otherwise, it seems unrealistic to expect that something like VBR at ~100 Mbps can perfectly replicate the original.

I’d love to hear your thoughts:

Is my assumption correct that lossless (or near-lossless) is the only way to truly preserve this kind of detail?

Are there specific encoder settings (AQ, 10-bit profiles, grain, etc.) that could help reduce artifacts in shadows without going fully lossless?

Thanks a lot in advance for sharing your knowledge – I’m eager to learn.

Comments

andyrpsmith wrote on 9/30/2025, 7:26 AM

How about uploading a short clip to a drive we can access (google gives you 15GB free) so we can try it. With the AV1 codec I was able to get 98.9% to the original with VBR of 53Mbit/s ave and 106Mb/s max.. Also what is the original media. Use the free mediainfo tool to upload the media info of your clip.

Rivas wrote on 9/30/2025, 7:36 AM

How about uploading a short clip to a drive we can access (google gives you 15GB free) so we can try it. With the AV1 codec I was able to get 98.9% to the original with VBR of 53Mbit/s ave and 106Mb/s max.. Also what is the original media. Use the free mediainfo tool to upload the media info of your clip.

Thanks a lot for your reply. I’ll prepare a short dark-scene gameplay clip and upload it to Google Drive as soon as possible, along with the MediaInfo summary.

I also realized I hadn’t paid enough attention to the difference between YUV 4:4:4 10-bit vs YUV 4:2:0 10-bit. That might explain some of the results I’ve been seeing.

Would it be an issue if I uploaded the test clip to YouTube instead of Google Drive? The reason I ask is that my final work is always intended for YouTube anyway. Even though I know YouTube will apply its own compression, my goal is to find the best way to maximize quality under those conditions. Since I usually upload in QHD (1440p), YouTube encodes my videos in VP9, which tends to look a bit cleaner than AVC.

By the way, when you mentioned achieving 98.9% fidelity to the original with around 53 Mbps average and 106 Mbps max bitrate — was that result achieved using Voukoder, or another encoder?

Thanks again for your help

Rivas wrote on 9/30/2025, 7:37 AM

How about uploading a short clip to a drive we can access (google gives you 15GB free) so we can try it. With the AV1 codec I was able to get 98.9% to the original with VBR of 53Mbit/s ave and 106Mb/s max.. Also what is the original media. Use the free mediainfo tool to upload the media info of your clip.

Original file information

General:

File size: 37.2 GiB

Duration: 55 min 08 s

Overall bitrate: ~96.5 Mb/s

Resolution: 2560×1440 (1440p)

Framerate: 60 fps

Format: AVC (H.264), High@L5.1, 8-bit, 4:2:0

Color: BT.709, Limited Range

Video stream:

Bitrate: ~96.1 Mb/s (max 100 Mb/s)

GOP: M=3, N=120

Reference frames: 4

Audio (main track):

Format: AAC LC, 2 channels

Bitrate: ~274 kb/s

Sample rate: 48 kHz

andyrpsmith wrote on 9/30/2025, 8:17 AM

I use Happy Otter Scripts render Plus (this uses the same FFMPEG as Voukoder). There is also a tool FFMetrics which you can use compare original and rendered files. Uploading to Youtube would not be a good example. I only need about 10sec not 55min. As the free Voukoder is no more Happy Otter Scripts is my alternative, it is not free but integrates well with Vegas. My media is XAVCS 4K 4:2:0 8Bit @100Mb/s

Rivas wrote on 9/30/2025, 10:13 AM

I use Happy Otter Scripts render Plus (this uses the same FFMPEG as Voukoder). There is also a tool FFMetrics which you can use compare original and rendered files. Uploading to Youtube would not be a good example. I only need about 10sec not 55min. As the free Voukoder is no more Happy Otter Scripts is my alternative, it is not free but integrates well with Vegas. My media is XAVCS 4K 4:2:0 8Bit @100Mb/s

By the way, if you ever need the free version of Voukoder, I keep a copy available on my Discord server (in the #voukoder channel under files). Here’s the invite link if you’d like access:
👉 https://discord.gg/mjHecFAE

Rivas wrote on 9/30/2025, 11:20 AM

I use Happy Otter Scripts render Plus (this uses the same FFMPEG as Voukoder). There is also a tool FFMetrics which you can use compare original and rendered files. Uploading to Youtube would not be a good example. I only need about 10sec not 55min. As the free Voukoder is no more Happy Otter Scripts is my alternative, it is not free but integrates well with Vegas. My media is XAVCS 4K 4:2:0 8Bit @100Mb/s

Here are the test files I rendered. I uploaded them to Google Drive:
👉 https://drive.google.com/drive/folders/1MswPHpE14cxyY4r75IsYUuUYz8npp9kW?usp=sharing

To make it easier to identify the settings, I included the main parameters in each filename. Here’s what each part means:

DYLB-Ep03 → Project/Game reference (Dying Light Beast, Episode 03).

HEVC / AV1 → Codec used (HEVC = H.265, AV1 = AOMedia Video 1).

VBR / CQP → Rate control method:

VBR = Variable Bitrate (target + max bitrate).

CQP = Constant Quantization Parameter (constant quality, file size may vary).

100M-150M → Target bitrate (100 Mbps) and maximum bitrate (150 Mbps).

QP15 / QP18 → Constant QP values (lower = higher quality, larger file size).

P4 / P7 → Encoder preset (P7 = slowest/best quality, P4 = faster but less efficient).

Main10 → Profile used (10-bit, 4:2:0 chroma subsampling).

YUV 420 / YUV 444 → Chroma subsampling (4:2:0 = standard for YouTube, 4:4:4 = full color resolution).

1pass → Single-pass encoding (as opposed to multipass).

HQ / AQ → Tune settings:

HQ = High Quality.

AQ = Adaptive Quantization enabled (distributes bits more efficiently in dark or complex areas).

RivasGPT → Just my internal tag.

Rivas wrote on 9/30/2025, 11:25 AM

I use Happy Otter Scripts render Plus (this uses the same FFMPEG as Voukoder). There is also a tool FFMetrics which you can use compare original and rendered files. Uploading to Youtube would not be a good example. I only need about 10sec not 55min. As the free Voukoder is no more Happy Otter Scripts is my alternative, it is not free but integrates well with Vegas. My media is XAVCS 4K 4:2:0 8Bit @100Mb/s

At the moment I’m at my office on the other side of the city and won’t be back at the studio until later. I just remotely accessed my machine and prepared the files, but I can’t personally make a proper visual evaluation right now because the video quality over TS remote access is too low.

andyrpsmith wrote on 9/30/2025, 11:37 AM

They all look to be rendered, I hoped for an unrendered file so I can render it to compare.

Rivas wrote on 9/30/2025, 11:46 AM

They all look to be rendered, I hoped for an unrendered file so I can render it to compare.

Got it, I’ll try to provide that later. I’ll record a short 1-minute clip with a very dark and high-motion scene.

john_dennis wrote on 9/30/2025, 11:56 AM

@Rivas Your original capture has a rather long group of pictures (GOP) and a fair distance between P-frames for acquisition.

GOP: M=3, N=120

It would certainly be easier on any output encoder if you would lower the GOP to 60, 30 or even 15 and get rid of the B-frames altogether.

My camera records at about the same final bit rate for UHD with this GOP structure.

Format settings, GOP                     : M=3, N=15

Rivas wrote on 9/30/2025, 1:58 PM

@Rivas Your original capture has a rather long group of pictures (GOP) and a fair distance between P-frames for acquisition.
GOP: M=3, N=120
It would certainly be easier on any output encoder if you would lower the GOP to 60, 30 or even 15 and get rid of the B-frames altogether.

My camera records at about the same final bit rate for UHD with this GOP structure.
Format settings, GOP                     : M=3, N=15

Thanks for the detailed explanation, John. I’ll run new tests keeping your suggestions in mind — especially lowering the GOP size to 60/30/15 and removing B-frames.