I am using the latest driver version available through GeForce experience. Regarding OBS, I have tried it, Bandicam, Fraps, and ShareX. Doesn't seem to impact the issue whatsoever. I should emphasize again that the issue only occurs when I manually increase the bit rate. If I use the default preset, visual quality is lower but not issues occur.

I am using the latest driver version available through GeForce experience.

That was not the question.
There are more tha 1 type of driver depending also on the use for it.
So I asked the exact number.

I have never manually installed drivers so I am not aware of other types of drivers. I just use GeForce experience to autoupdate my graphics card drivers when a new one comes out. The current version is: 457.09 Released on: 10/29/2020

That one is good, but better for your are the by Vegas development suggested Studio drivers for working with Vegas you will find here for your GPU: https://www.nvidia.com/Download/driverResults.aspx/166106/en-uk

I'll try it out and get back to you.

I tried exporting the the studio driver and it didn't help

1. See if this sheds light on your driver question:

https://www.vegascreativesoftware.info/us/forum/vegas-pro-17-bandicam-hevc-files-unsupported--116964/#ca728341

2. Apparently Bandicam can record either CFR or VFR. Variable Frame Rate is unsupported in Vegas Pro 17, but should work in Vegas Pro 18.

3. Lossy files produced at such incredibly high bitrates are unlikely to play locally without a lot of blocking and pixelization caused by system lag. The fact that you say Youtube transcodes them correctly is a big giveaway that this is indeed the cause of your first issue. Make a smaller file for local playback.

4. If, as it seems, you are looking for the most accurate screen and motion reproduction, and generational integrity for uploading, machine encoding (NVENC) at the capture or render steps are exactly the wrong way to go. Since you say you have OBS, start here with the lossless UT software codec, and render to software HEVC in Vegas, or x265 if you also have Happy Otter Scripts or Voukoder. Feel free to post back with results.

I'll try HEVC thanks. However, I didn't excatly undestand the third point. Maybe I didn't explain it properly but my issue is that if I increase the bitrate while rendering I get the weird blockiness and pixelation. Both on my machine and Youtube. If I don't increase the bitrate I end up with quality lower than the original.

Maybe I didn't explain it properly but my issue is that if I increase the bitrate while rendering I get the weird blockiness and pixelation.

- Uncompressed Media with too low birtate become sometimes better with higher bitrates.
- When a codec is already used to compress the video the bitrate does not need the uncompressed values, but
- if the modern codec AVC is used the bitrate can be lower and with the todays max compression with
- HEVC the bitrate can be much lower for a good picture,
- because that is the task for the compressor, sometimes with help of a GPU.

alright, this a lot to take in. So what you suggest is that I change the recording codec to remove the blockiness or is there something else?

Don't know.
I only use Screenpresso and/or Vegas Screencapture on laptop and the same on desktop plus sometimes OBS to not use too much of my NV encoder while doing a 4K screenvideo with Vegas editing

One small update, I tried exporting with the exact same settings in the trial version of Premiere Pro and it worked just fine. So I assume there is no issues with the original file. The problem is probably with my Vegas settings

Both on my machine and Youtube. If I don't increase the bitrate I end up with quality lower than the original.

OK, your hardware encoding bitrate is set too high. Use software.

Where can I switch from hardware encoding to software?

@Szerhij-Taranenko

"Where can I switch from hardware encoding to software?"

@Szerhij-Taranenko

Where can I switch from hardware encoding to software?

If you mean here in Vegas @john_dennis showed you.
If you mean in OBS, look here

Okay, a major update. I have tried recording with OBS and a different codec as you have outlined and it hasn't really impacted my issue. Also, I tried using software encoding and what happens is that Vegas uses 99% of my ram (64GB, that is) and crashes about half way through.

......what happens is that Vegas uses 99% of my ram (64GB, that is)

Where do you see that ?
And give also a screenshot of Options/Preferences/Videoat your settings

I see that by just monitoring my task manager while it's rendering. I used excatly the same setting as john_dennis outlined in the screenshot above.

That's what I thought and is good, but that may not use all your Memory.
Thatswhy I asked you to give a screenshot of your settings in the program Vegas Pro 17 concerning what is in the menu Options/Preferences/Video

@Szerhij-Taranenko

Though I never use those settings and didn't really recommend them, they are working for me.

I only showed them as an example of how to accomplish what you said you want to do.

It feels like the problem is with the framerate due to the nature of the issue. On the other hand, I don't understand why the same process works in other programs just fine. I'll try messing around with the setting some more.