if possible set this to 0

I'll try, thank you!

I'm sorry, it's a little different. The video renders normally, but at one point the image turns black. At that point, memory usage begins to increase relatively quickly.

I set Dynamic RAM Preview Max to 0, but unfortunately that didn't help.

I don't understand this. There is no other application to fill the RAM, but the memory is full. But Vegas doesn't show full memory.

There is nothing to fill the memory, yet the memory is full and frames is black:

No idea what is wrong at you, can be hardware related, Windows version and -settings related and so on.
This how a render ( as yours I prosume) goes with me on laptop with VPro 18. Processor is very high due to also making a screenvideo.

I don't have a specially modified system. The rendering process is similar to yours. But at one point (randomly) it goes crazy during rendering. Rendering turns black and RAM fills up.

Are you using an fx anywhere? If yes, which ones and have you tried disabling them to see if they're the culprit?

I wonder if it related to GPU onboard RAM what GPU are you using?

GTX 1080 8GB VRAM
fx i used only Levels(RGB comp to studio preset) on master output

I assume you are using a GPU accelerated render template? Does it also do this with a CPU only render? Also try setting the number of rendering threads to either the number of CPU cores or twice that number if your CPU has hyperthreading.

This software declares GPU support, I will not use only CPU for rendering. I think it's a problem with supporting this format from the camera. Vegas often has a problem with some mp4 files, but I want to solve this. My customer will not wait half a year for an update, which may not even solve it.
My OS is fine, the hardware is standard (i7 4790k, GTX 1080, 16GB RAM). God, in this case I'm rendering 720p, not 8K video.

I rendered this resolution just fine a few years ago with 6GB of RAM and Sony Vegas 12 😐

Vegas Pro 18 is great, but this is a weird problem.

I like to help you because on my laptop has approximately the same hardware with a little weaker GPU and a a little better CPU (7th generation i7) and there the software goes without problems.
So I like to try a little project of you that has the error.
Can you post such a veg included sourcefiles on a cloudserver we can download it with a link you give here? You may also send me that link on PM.

Try turning "GPU acceleration of video" OFF, when you go to encode. You can leave video decoding on, and still use NVENC or QSV hardware gpu encoding. See if that fixes the high memory fault

You can mark that as the solution in case others have the same issues. I remember another thread where someone also had to disable an unsupported older Intel integrated GPU.

@AlesCZ
Did you use the latest driver for that Intel 4600, i.c. 15.40.46.5144 from aug 18 2020?

New info: For a project with my YI 4K + camera, I must to turn on the GPU on the processor, otherwise Vegas causes artifacts(black frames, blinking frames....), or it gets stuck and crash during editing.