Gigabit networking

vicmilt wrote on 6/19/2007, 3:59 AM
I recently upgraded my system to a new computer -Q6600 with SATA RAID plus 4 SATA "outboard" media drives.
My old workhorse is an Pentium P920. It's really not too shabby, and I didn't want to totally discard it.

So I installed Gigabit network cards in both machines and also upgraded both machines to use SATA drives with the following observations:

A - Everything runs much more smoothly (but still not perfectly all the time, and of course renders are quicker)
B - I can work on the slower P920 as an outboard terminal, utitlizing media on the SATA drives on the Q6600. The movies play equally well on either machine. This has saved untold hours of "sneaker netting".
C - I can digitize new HDV footage to the SATA drives on the Q6600 using either machine as the input terminal.

The advantage to all this? Well, for starters, I have not put ANYTHING on the new Q6600 but Vegas and DVD Architect. I do all my photoshop work on the older machine and save directly to one of the data drives on the newer machine. Also, on occasion, I have an assistant logging new footage and doing simple edits utilizing the media on the Q6600 via the P920, while I am actually editing on the Q6600 - but NOT using the same files.

Mainly, it's nice to have all the media "locked down" in one place and to be able to use it across the network. I have mapped all the Q6600 drives to the desktop of the P920. It's all seamless and invisible now.

If anyone has any suggestions of other ways to use this newfound strength, I'd appreciate in hearing about it, and the main thing is... don't throw away your older units.

best,
v

Comments

farss wrote on 6/19/2007, 5:14 AM
Did you upgrade all your hubs?
Unless your hubs are also Gigabit you're not getting anything like the full 1GB/sec. Well actually once you get upto that speed what you really want are Switches, not hubs although I think most gigabit hubs are switches anyway.

Also if your cabling is long it can pay to also upgrade that.

I've recently purchased a Thecus 5200 and fitted 5 x 400GB drives although you might find 5 x 500GB a better bang for the dollar. Anything we think we'll need outside of a current project we keep on the Thecus box. Have a dedicated box for storing common files is a plus, we leave it on 24/7 as it's not very power hungry.

Bob.
JJKizak wrote on 6/19/2007, 5:23 AM
And don't forget the new network cat 6 cables. One of my guru buddies told me they were kind of stuck on cat 6 because they couldn't wind the wires together any tighter or they would break. He installs all over the USA.
JJK
Tech Diver wrote on 6/19/2007, 6:58 AM
Cat 6 is not required. Cat 5e fully supports Gigabit.
2G wrote on 6/19/2007, 7:17 AM
I am SO glad this discussion has gotten started. I set up a gigabit network a couple of years ago specifically for the purpose you described of sharing the raw video for projects across multiple computers. Unfortunately, my transfer rate has been significantly lower than what I would have expected.

Setup: all the network cards and switch are Gigabit; I use Cat-6 cables; the computers and the switch are all side by side, and the cables are about 3ft long. All of the computers are running up to date XP and are all reasonable speed processors. So theoretically, there is nothing hardware related. (I even replaced the switch with a different brand just in case, but same results).

Bottom line is that on a completely idle network, I max out at around 200MB transfer (20% utilization of a gigabit network). I'm a software engineer by trade, and I completely understand packet overhead. I wouldn't expect 100% gigabit speed on data transfer. But I would expect a lot more than 20% (and it's usually 12%-15%).

I've investigated using jumbo packet options, etc. But still little success.

My network is faster than the standard 100MB network. So I know I'm using the gigabit function.

I have been asking on forums, etc. for a year for somebody to tell me that a) they get the same thing or b) they get a lot better. I have talked to the switch mfg, and they gave me the runaround and didn't really provide any info.

If at all possible, could I get you to do one very simple test on your network? Could you do a simple copy of a large (2-3GB) file from one computer to another on your gigabit network, and time the transfer? It needs to be large enough so that it can be timed in seconds or minutes. Also, at the same time, open the task manager to the network tab, and note the utilization percentage of the network card during the transfer. Just give me the file size and the number of seconds it took to copy it, along with the average network utilization from the task manager. That will be extremely helpful to me.

If you're getting what I'm getting, then I'll drop it and move on to some other dragons to slay. But if your numbers are significantly different than mine, then I've got more research ahead.

Thank you so much.

2G
farss wrote on 6/19/2007, 7:43 AM
10,216 Mbytes took 7:50.
Didn't check network utilization.

A couple of things to keep in mind, the Intel Pro/1000 network cards seem to be the best of the bunch for jumbo packets. Disk I/O speeds and CPU speed are also perhaps entering into the mix.
douglas_clark wrote on 6/19/2007, 8:21 AM
I copied a 5153 MB avi from main PC (SATA2) to laptop (ATA100, 7200 rpm)
over gigabit LAN -- time 4:43. 5153 MB/283 s = 18.2 MB/s
....times 8 = 146 Mbps, which is 14% of theoretical 1 Gbps.

farss' transfer was 10216 MB/470 s = 21.7 MB/s.

Home-built ASUS PRIME Z270-A, i7-7700K, 32GB; Win 10 Pro x64 (22H2);
- Intel HD Graphics 630 (built-in); no video card; ViewSonic VP3268-4K display via HDMI
- C: Samsung SSD 970 EVO 1TB; + several 10TB HDDs
- Røde AI-1 via Røde AI-1 ASIO driver;

JJKizak wrote on 6/19/2007, 9:28 AM
Unrelated but the fastest transfer rates I ever got were from a SATA raid to USB2 external drive. It blew away my network which is all 1 gig.
So I transfer from one computer to the external drive then plug the drive into the other computer. Much faster than the network.
JJK
rmack350 wrote on 6/19/2007, 10:11 AM
Just some food for thought.

I remember reading an article a few years ago about gigabit ethernet on intel chipsets and there was a comment that it's enough to swamp a PCI bus. That was the article's reasoning for putting the ethernet controller on the northbridge at that time. (They're all back on the south bridge these days).

Bob has pointed out a file server that uses aggregated Gig-ethernet for file transfers. This is a system that gangs adapters together to get higher throughput. I'm going to guess that people dealing with aggregation will have more info for you about throughput.

Ethernet usually doesn't gaurantee throughput. You get what's available, and this is why it's not usually chosen for high bandwidth realtime video playback. Happily, DV and HDV aren't really "high bandwidth", and many of the things you'd do over a gigabit network aren't realtime anyway, so it works out.

And of course you need a switch. Hubs broadcast to everyone on the segment and reduce the overall throughput.

Rob Mack
rmack350 wrote on 6/19/2007, 10:35 AM
The NAS product Bob had mentioned is described here:
http://www.xdt.com.au/Products/filmSTOR/

I'm not pushing this, but it seems like a good jumping-off point to start asking useful questions about Gb network performance.

Rob Mack
douglas_clark wrote on 6/19/2007, 10:53 AM
Vic: ... I have mapped all the Q6600 drives to the desktop of the P920. ...

I assume you mean you mapped all your drives/partitions to the same drive letters on both machines, right? Or do you actually have them as subdirectories on your desktop?

Other thoughts....
- I have a dual monitor setup on my main PC, and the monitors have dual inputs. So I have my "extra PC" connected to the second input on one screen. I just press the input selector button on the monitor to switch to the other PC's screen. Alas, I still have a second keyboard and mouse to fumble with.

- I do my office work and surfing on my laptop, which is also my on-location recorder. No office apps on my media PC.

- Media collections are on the main PC: stock music, stock photos and stock video. All accessible from my 3 PCs...same drive letters on each PC. I also have all my software in an "Install" directory on the main PC.

- The main problem is backing all this stuff up. Much of the stock media is on CD & DVD, but mixed together with stuff that has been downloaded. Anyone have a good idea how to distinguish stuff that doesn't need backing up as carefully, since I have the original on tape or dvd? I'm ready for a holographic disk ;-)

Douglas

Home-built ASUS PRIME Z270-A, i7-7700K, 32GB; Win 10 Pro x64 (22H2);
- Intel HD Graphics 630 (built-in); no video card; ViewSonic VP3268-4K display via HDMI
- C: Samsung SSD 970 EVO 1TB; + several 10TB HDDs
- Røde AI-1 via Røde AI-1 ASIO driver;

LReavis wrote on 6/19/2007, 12:24 PM
you can use just one keyboard and one mouse for 2 computers if you have 2 monitors - with the help of Synergy (http://synergy2.sourceforge.net/). Just choose one of the machines to be the host and one the client, then network both together and your mouse pointer will slide seamlessly from the monitor on the left (let's say it's the client) to the monitor on the right (connected to the host computer). The keyboard will be directed to whichever computer's monitor the mouse pointer is on at the moment.
This system works flawlessly except when rendering ties up 100% of the CPU; then you'll get stuck on whichever monitor you're on. In those cases, I use a backup USB KVM (keyboard, video, mouse) switch to manually switch back to the other computer (the KVM switch also switches the second monitor so that I have 2 monitors for my main computer).

Incidentally, Synergy is free.
apit34356 wrote on 6/19/2007, 2:00 PM
Couple of simple points for large fast transfers between two machines; avoid network switches, connect direct. If you have more that three machines add additional network cards permitting direct connects. One can have a common network card with direct connect lan cards for easier general network work. On all the direct connects cards, remove all excessive IO management that is not file oriented. Increase memory if you can, increase disk caching. An old forum member called "Jim" a few years pushed network lan's fairly hard for network rendering, I think he's a mod with DSE on DMN vegas forums(?), but he would be an excellent source for the do's and don'ts.
Xander wrote on 6/19/2007, 2:48 PM
Gigabit Ethernet is a layer two technology. IP is a layer three technology. File transfer is an application layer technology. As much as GE helps layer 2 and jumbo frames help layer 3. Window size of FTP and the software used help throughout the most. Remember applications need CPU cycles.
apit34356 wrote on 6/19/2007, 3:13 PM
Xander, I was trying to keep it simple, but what I was suggesting was reducing the number transfer protocols that sometimes are added when not needed , ie, IPX ( this protocol has overhead issues). The end goal was as you stated,"applications need CPU cycles."
Tech Diver wrote on 6/20/2007, 9:55 AM
There is a well-known free network performance test tool called Qcheck, that is from a company called Ixia. You have to supply your name and email to download it, but no one will contact you or share your info. The pointer to their site is:
http://www.ixiacom.com/products/display?skey=qcheck

I have have acheived through-puts between 400Mbps and 600Mbps on my Gigabit network. Such numbers are considered good. Remember, not all Gigabit NICs are FULL duplex. Also, for those who use jumbo frames, Windows file sharing does not work correctly. Also, ALL your network devices must support jumbo frames if you go that route.