The bottleneck is the network connection. Whenever you switch back to Vegas the media pool is refreshed, causing all the files to be scanned for thumbnail images again. When reading from a local hard drive this can take a while. Fetching the thumbnail images from files across a network can be painfully slow.

You might want to try flipping "Options / Preferences / General / Close media when not the active application" to off. This will keep the files open in Vegas when you switch to another program and then the media pool may not have to be refreshed when switching back. The downside is that you may be unable to edit any of the media files or use them in other applications while Vegas is open. The computer that hosts the media files may also be bogged down more by keeping all the network connections open all the time.