When it comes to lowering the file size, the most important thing is bitrate. Bitrate is the amount of data per second encoded in a clip. So if you have 8mbps (the standard for YouTube 1080p), a 1 hour video should be around 28,800 megabits or 3.6GB. If you lower the bitrate to 4mbps, your file should be half the size. The problem with this is compression.

If you lower the bitrate without lowering the resolution, for instance, then you will half the information in the same size frame which means more compression artifacts. This would then be a lower quality image. However if you know that people will be viewing this video primarily in a windowed mode (like a YT player, not in full screen) or on a phone, you can likely get away with 720p or even 480p. This way you can lower your bitrate without introducing tons of compression artifacts. 1080p is roughly 2x the size of 720p which is roughly 2x the size of 480p. This means that you can go from 1080p 8mbps to 720p 4mbps without increasing the compression of the video. If most people are going to be watching full screen on a 1080p monitor than you will definitely notice a difference, however if you watch it in a 720p window the quality should be roughly the same with a 720p file or 1080p file.

Vegas gives you an estimated file size in the render window. I would recommend messing with the bitrate (bps) in either the AVC/AAC format or the HEVC/AAC format (both of which use .mp4 containers). When you have the estimated file size to something you consider reasonable, render and watch it back. Is the quality reasonable to you? If not, increase the bitrate until you find the sweet spot between quality and file size. I typically use the YouTube recommended bitrates when uploading to the internet, but your mileage may vary.