This is a vast oversimplification, so keep in mind that the actual process is probably a lot more complex.

Imagine, if you will, that the video frame is a checkerboard of red & black squares. It's also 8x8 squares. If, for whatever reason, you now need a 4x4 board, you would have to resize it. The simplistic and fast way to do this would be to skip every other square in both directions and keep only the ones you don't skip. So, for example, you might keep squares 1, 3, 5, and 7 in rows 1, 3, 5, and 7. Now, if you look at these squares specifically you might that they are all black. This means that your resulting 4x4 checkerboard is all black. It's very fast to do this, but the result isn't really what you wanted. This is fast resizing. In actual practice it's not as bad as this horrid example, but you can see the point. Information is simply thrown away in order to get the result as quickly as possible.

Instead of chucking stuff out, you could combine adjacent squares that will end up occupying the same new square. In the new checkerboard, square 1 of row 1 is made up of squares 1 & 2 from both rows 1 & 2 of the original. This means that you'll blend two black squares and two red squares together to get a single maroon square. It takes time to do this averaging so it's not as fast. However, the result might be more faithful to the original.

Of course, this checkerboard scenario isn't very realistic. Video is 720x480 pixels or so. It's very seldom you will have a video frame that consists of regular alternating pixels of high contrast colors. The details of the image tend to cover larger groups so using either method may or may not produce better results. Fast resizing may be sharper but look blocky. Averaging my look smoother but may also look blurry. It really depends on what the original image looks like. There are also different methods of averaging that take into account more surrounding pixels, giving them less weight in the average. This can improve smoothness at the cost of losing more sharpness.

The less averaging that is done, the faster the process will go. That will hold true whether the end result would benefit and look better from averaging or not. So, Fast Video Resizing will always render faster than not using it. Whether it looks better or not depends.