NTSF won't cause file size problems. Under FAT 32 you are going to run into the brick wall of the 4GB file size limitation very quickly if you create uncompressed files like AVI. There's no way around it if you need a single file that's larger unless you use a newer file system that supports larger files like NTFS does.

Under Windows98SE Video Factory will gracefully create as many files as it takes limited to your free space on your drive. However you'll end up with myfile, myfile-a, myfile-b and so on.