Affordable 10G Network for Small Studios...

TheRhino wrote on 5/21/2019, 9:24 AM

10G networking is becoming more affordable. To sustain 4K workflow, I edit on my fastest 9900K workstation & render the most CPU-intensive output there while allowing my 2 other (slower Xeon) workstations to render various other client-requested files. A good example is when the client still wants Blu-rays & DVDs , wants different text messages in the lower 3rd for various audiences, allowing Mercalli 4 stand-alone to batch-render all-day w/favorite settings, etc... This requires having copies of the source video & VEG files on each workstation. Previously I had (2) Mellanox 10G PCIe cards in my main editing workstation & (1) card in each of the render workstations, but that setup had limitations.

Then I found a 10G switch for <$160 here:  https://www.amazon.com/gp/product/B07HYKQFWD
(MikroTik 5-Port Desktop Switch)
Paired with $30 Mellanox PCIe cards found here: https://www.amazon.com/gp/product/B016OYD0D4
You can connect several workstations & a NAS in a small studio environment
For short runs use this: https://www.amazon.com/gp/product/B00U8BL09Q
($10 - $20 10G SFP+ DAC Cable)

I have a QNAP NAS mounted in a server rack in a back room.
To go the distance get these: https://www.amazon.com/gp/product/B00U8Q7946
($21  10 Gigabit SFP+ LC Multi-mode Transceivers)
And this https://www.amazon.com/gp/product/B07DP766QG
($20-$40 10Gb 50/125 Multimode Duplex Fiber Optic Patch Cable)

6-5-2019 EDIT: QNAP's new affordable 10G switches (QSW-308 & QSW-303s) are arriving soon, price TBA. Meanwhile their older 8-Port is discounted to $359 here: https://www.bhphotovideo.com/c/product/1413712-REG/qnap_qsw_804_4c_us_qsw_804_4c_8_port_unmanaged_10gbe.html

I routinely get 500-600 MB/s transfer speeds between workstations, mainly because that is the fastest speed of my current SATA RAID0 drives connected to PCIe 2.0 hardware controllers on my older Xeon workstations...

Last changed by TheRhino

Workstation D with $1,350 USD of upgrades in April, 2019
--$500 9900K @ 5.0ghz
--$140 Corsair H150i liquid cooling with 360mm radiator (3 fans)
--$200 open box Asus Z390 WS (PLX chip manages 4/5 PCIe slots)
--$160 32GB of G.Skill DDR4 3000
--$350 refurbished, but like-new Radeon Vega 64 LQ (liquid cooled)

Renders Vegas11 "Red Car Test" (QSV) in 13s-14s when clocked at 5.0 ghz
(note: BOTH onboard Intel & Vega64 show utilization during QSV renders...)

Source Video1 = 4TB RAID0--(2) 2TB M.2 on motherboard in RAID0
Source Video2 = 4TB RAID0--(2) 2TB M.2 (1) via U.2 adapter & (1) on separate PCIe card
Target Video1 = 32TB RAID0--(4) 8TB SATA hot-swap drives on PCIe RAID10 card

10G Network using cheap Mellanox2 Adapters
Copy of Work Files, Source & Output Video, OS Images on QNAP 653b NAS
Blackmagic Decklink PCie card for capturing from tape, etc.
(2) internal BR Burners connected via USB 3.0 to SATA adapters
Old Cooler Master CM Stacker ATX case with (13) 5.25" front drive-bays holds & cools everything.

Workstations A, B & C are 6-core 4.0ghz Xeon 5660 or I7 980x on Asus P6T6 motherboards.

Comments

Kinvermark wrote on 5/21/2019, 4:20 PM

Thanks for contributing that info.! Nicely laid out. That's a lot cheaper than it was just a year ago.

Musicvid wrote on 5/22/2019, 8:11 PM

Yes, I agree, that post is well laid out, and very thoughtfully presented.

That said, 10 Gigabit LAN sounds like a lot of bandwidth. I don't think most editors will ever need that kind of horsepower, kind of like driving a dragster to work. But 8k is coming...

As for RAID 10, it's best to check your throughput and error count against RAID 0 and even JBOD. Really, I would opt for 4 times the available storage, not 4 times the chance of errors if even one drive glitches.

fifonik wrote on 5/23/2019, 2:36 AM

I routinely get 500-600 Mb/s transfer speeds between workstations, mainly because that is the fastest speed of my RAID10 drives... 

Are there typo in speed? Did you mean MB/s instead?

I'm having 115+ MB/s (that is 920 Mb/s) on my integrated in MB 1 Gbps NIC with a bit old (and already not supported) NAS (RAID5) with 1 Gbps NIC (no link aggregation).

P.S. Just curious, what drives do you have in RAID? Cannot think of any RAID ready HDD that give that speed (if speed is 500+ MB/s). And it is a bit low for modern SSD.

Last changed by fifonik on 5/23/2019, 3:12 AM, changed a total of 3 times.

Camcorder: Panasonic X920

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 3700X, RAM: G'Skill 16 GB DDR4 3200, Graphics card: MSI RX470 4GB, SSD: Samsung 850 EVO SATA 250MB, HDDs: Seagate & Toshiba 2TB, OS: Windows 10 64-bit 1903

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17

TheRhino wrote on 5/23/2019, 7:45 AM

Yes, I agree, that post is well laid out, and very thoughtfully presented.

That said, 10 Gigabit LAN sounds like a lot of bandwidth. I don't think most editors will ever need that kind of horsepower, kind of like driving a dragster to work. But 8k is coming...

As for RAID 10, it's best to check your throughput and error count against RAID 0 and even JBOD. Really, I would opt for 4 times the available storage, not 4 times the chance of errors if even one drive glitches.

The last 4K client project that I had to move between workstations was 2TB, so with 10G the files transfer over lunch versus overnight... On my new 9900K workstation, the SOURCE files are on a (2)x2TB M.2 Intel 660p in RAID0 and the TARGET is a (2)x2TB M.2 Intel 660p in RAID0. I don't have any M.2 drives on the other workstations so their SOURCE drives are a (6) 2TB SATA drive RAID0 and TARGET are (6) 8TB SATA drive RAID10. Empty, the SATA RAID0 can write at about 700 MB/s, but it can slow to 500 when 4K files are transferring...

I have about (40) 8-10 year-old 1-2 TB SATA drives and (16) newer $150 8TB WD RED drives shucked from WD cases... Of the original 40, 2 died under warranty & 2 died in the last couple years - running much hotter now churning-out 4K... (10% failed in 10 years...) I am slowing disconnecting the old drives for offsite storage...

When RAID10 fails I can keep working & just plug-in the spare into the hot-swap trays that are externally accessible. A 2 minute fix. When RAID0 fails, I toss the bad drive, reformat the RAID, and pull the files from either another workstation or the NAS losing a half-day vs. 2 minutes... 10G networking throughout will make RAID10 less important, but I still like it... Having (3) workstations means I always have something to do vs. waiting on file transfers, renders, etc. IMO I stay competitive by offering high quality editing, in less time, and for less cost than competitors...

To date, crossing fingers, I have only lost an hour here & there because I forgot to backup my most recent VEG draft... The MOST common mistake is for me to write an older VEG over a newer VEG even though I name them draft1, draft2, etc... Woops, we all make mistakes & live & learn!

Last changed by TheRhino on 5/23/2019, 5:14 PM, changed a total of 1 times.

Workstation D with $1,350 USD of upgrades in April, 2019
--$500 9900K @ 5.0ghz
--$140 Corsair H150i liquid cooling with 360mm radiator (3 fans)
--$200 open box Asus Z390 WS (PLX chip manages 4/5 PCIe slots)
--$160 32GB of G.Skill DDR4 3000
--$350 refurbished, but like-new Radeon Vega 64 LQ (liquid cooled)

Renders Vegas11 "Red Car Test" (QSV) in 13s-14s when clocked at 5.0 ghz
(note: BOTH onboard Intel & Vega64 show utilization during QSV renders...)

Source Video1 = 4TB RAID0--(2) 2TB M.2 on motherboard in RAID0
Source Video2 = 4TB RAID0--(2) 2TB M.2 (1) via U.2 adapter & (1) on separate PCIe card
Target Video1 = 32TB RAID0--(4) 8TB SATA hot-swap drives on PCIe RAID10 card

10G Network using cheap Mellanox2 Adapters
Copy of Work Files, Source & Output Video, OS Images on QNAP 653b NAS
Blackmagic Decklink PCie card for capturing from tape, etc.
(2) internal BR Burners connected via USB 3.0 to SATA adapters
Old Cooler Master CM Stacker ATX case with (13) 5.25" front drive-bays holds & cools everything.

Workstations A, B & C are 6-core 4.0ghz Xeon 5660 or I7 980x on Asus P6T6 motherboards.

fifonik wrote on 5/23/2019, 4:06 PM

Could you advice, how many disks (WD Red 8TB) do you have in RAID10? It does not looks even theoretically possible to have 500-600 MB/s read speed for 4 disks RAID10 setup. It only doubles the speed of a single drive that is by specs 210 MB/s peak (at the outer cylinders), linear is lower. So RAID10 for 4 disks setup should not be able to show speed higher than 420 MB/s. In practice it is even slower, I'd expect ~350 MB/s max.

Camcorder: Panasonic X920

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 3700X, RAM: G'Skill 16 GB DDR4 3200, Graphics card: MSI RX470 4GB, SSD: Samsung 850 EVO SATA 250MB, HDDs: Seagate & Toshiba 2TB, OS: Windows 10 64-bit 1903

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17

TheRhino wrote on 5/23/2019, 5:13 PM

On the SAME workstation...
SOURCE drives are a (6) 2TB SATA RAID0 (w/faster drives than the REDs...)
TARGET drives are a (6) 8TB SATA RAID10
They are connected to an older LSI PCIe 12-drive RAID controller...

I need to copy source files from one workstation to the other, not the target files... I typically copy the source files between workstations before I ever start a project...

Last changed by TheRhino on 5/25/2019, 8:16 AM, changed a total of 2 times.

Workstation D with $1,350 USD of upgrades in April, 2019
--$500 9900K @ 5.0ghz
--$140 Corsair H150i liquid cooling with 360mm radiator (3 fans)
--$200 open box Asus Z390 WS (PLX chip manages 4/5 PCIe slots)
--$160 32GB of G.Skill DDR4 3000
--$350 refurbished, but like-new Radeon Vega 64 LQ (liquid cooled)

Renders Vegas11 "Red Car Test" (QSV) in 13s-14s when clocked at 5.0 ghz
(note: BOTH onboard Intel & Vega64 show utilization during QSV renders...)

Source Video1 = 4TB RAID0--(2) 2TB M.2 on motherboard in RAID0
Source Video2 = 4TB RAID0--(2) 2TB M.2 (1) via U.2 adapter & (1) on separate PCIe card
Target Video1 = 32TB RAID0--(4) 8TB SATA hot-swap drives on PCIe RAID10 card

10G Network using cheap Mellanox2 Adapters
Copy of Work Files, Source & Output Video, OS Images on QNAP 653b NAS
Blackmagic Decklink PCie card for capturing from tape, etc.
(2) internal BR Burners connected via USB 3.0 to SATA adapters
Old Cooler Master CM Stacker ATX case with (13) 5.25" front drive-bays holds & cools everything.

Workstations A, B & C are 6-core 4.0ghz Xeon 5660 or I7 980x on Asus P6T6 motherboards.

Musicvid wrote on 5/23/2019, 5:42 PM

So what is the MediaInfo for your SOURCE files?

https://www.vegascreativesoftware.info/us/forum/faq-how-to-post-mediainfo-and-vegas-pro-file-properties--104561/

The math is still sounding like you have more horsepower than needed for optimal 4k transfer. What are your transfer rates when configured as JBOD? What about 4x the storage space instead of the redundancy? Striping is not usually considered a big player when moving video files around.

Drive throughput historically is not a bottleneck, unless you are transferring over SDI.

Oh, I still admire your setup!

 

 

fifonik wrote on 5/23/2019, 8:01 PM

@TheRhino Thanks. If I understand correctly, data was read from RAID0 (6 * 2TB fast HDDs) on remote workstation through 10G network and written to RAID10 (6 * 8TB WD RED) on local workstation. Is it correct?

@Musicvid JBOD will not give 4x more space against RAID10. 2x only (for 4 drives RAID). I agree, that drive would not be bottleneck until editing multi-cam uncompressed. The setup will be nice to have while copy big amount of data is needed (definitely not my user case).

Last changed by fifonik on 5/24/2019, 1:00 AM, changed a total of 1 times.

Camcorder: Panasonic X920

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 3700X, RAM: G'Skill 16 GB DDR4 3200, Graphics card: MSI RX470 4GB, SSD: Samsung 850 EVO SATA 250MB, HDDs: Seagate & Toshiba 2TB, OS: Windows 10 64-bit 1903

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17

TheRhino wrote on 5/24/2019, 8:48 AM

The SOURCE files on all workstations are on the faster RAID0 drives. (The 9900K has M.2 in RAID, the others are 6 2TB SATAs...) They are what needs to be copied from workstation-to-workstation. (I do not try to load files across the network into Vegas, copies are always stored locally....) Once they & the VEG files are copied, I render whatever files the client has requested to the TARGET RAID10 & also store the VEG files on RAID10 so I do not lose any work progress if a drive should fail... Although only 2/40 drives have failed in 10 years, they failed when I was meeting deadlines. RAID10 meant that I could just keep working & did not have to re-render anything. I was able to meet expedited delivery dates for which I charge a premium. The premium more than pays for building & maintaining the RAID10s...

For instance, I got the 8TB WD REDs for only $150 each & they are now even lower... That's 16TB in RAID10 for only $600... BTW for US folks the shuckable WD 8TB Easystore drives are on sale again at Best Buy (& BB ebay) for $130 USD. I take the RED drives out of them & use them inside my NAS & RAID10s. I then fill the cases with the 8-10 year-old 1TB & 2TB drives, copy old client projects to them & place them in secure/offsite storage. This way if there is theft, fire, etc. I have not lost that work...

Last changed by TheRhino on 5/24/2019, 9:13 AM, changed a total of 4 times.

Workstation D with $1,350 USD of upgrades in April, 2019
--$500 9900K @ 5.0ghz
--$140 Corsair H150i liquid cooling with 360mm radiator (3 fans)
--$200 open box Asus Z390 WS (PLX chip manages 4/5 PCIe slots)
--$160 32GB of G.Skill DDR4 3000
--$350 refurbished, but like-new Radeon Vega 64 LQ (liquid cooled)

Renders Vegas11 "Red Car Test" (QSV) in 13s-14s when clocked at 5.0 ghz
(note: BOTH onboard Intel & Vega64 show utilization during QSV renders...)

Source Video1 = 4TB RAID0--(2) 2TB M.2 on motherboard in RAID0
Source Video2 = 4TB RAID0--(2) 2TB M.2 (1) via U.2 adapter & (1) on separate PCIe card
Target Video1 = 32TB RAID0--(4) 8TB SATA hot-swap drives on PCIe RAID10 card

10G Network using cheap Mellanox2 Adapters
Copy of Work Files, Source & Output Video, OS Images on QNAP 653b NAS
Blackmagic Decklink PCie card for capturing from tape, etc.
(2) internal BR Burners connected via USB 3.0 to SATA adapters
Old Cooler Master CM Stacker ATX case with (13) 5.25" front drive-bays holds & cools everything.

Workstations A, B & C are 6-core 4.0ghz Xeon 5660 or I7 980x on Asus P6T6 motherboards.

Musicvid wrote on 5/24/2019, 9:24 AM

If you wish to share your source properties, there is a link above. If not, that's ok, too.

Congratulations on a fine rig, which should meet or exceed your needs for years to come.

fifonik wrote on 5/24/2019, 6:20 PM

At the beginning it was about "500-600 Mb/s from HDDs in RAID10"

Now it is "500-600 MB/s from M.2 SSDs in RAID0"

These are two very different scenarios.

2 modern SSD M.2 in RAID0 should definitely give speed faster that 500 MB/s so the limitation is network in this case (my Samsung 850 EVO shows 550 MB/s, not M.2, not PRO, not in RAID0).

So, I still do not get what exactly did you test.

NVM, your setup looks extremely powerful and the only important thing here -- that you are happy.

Last changed by fifonik on 5/24/2019, 6:24 PM, changed a total of 1 times.

Camcorder: Panasonic X920

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 3700X, RAM: G'Skill 16 GB DDR4 3200, Graphics card: MSI RX470 4GB, SSD: Samsung 850 EVO SATA 250MB, HDDs: Seagate & Toshiba 2TB, OS: Windows 10 64-bit 1903

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17

john_dennis wrote on 5/25/2019, 12:42 AM

"...allow 2 other workstations to render various client file requests vs. rendering in the background like I did with 1080p... This requires having copies of the source video & VEG files on each workstation."

Musicvid and fifonik. You're getting too far down into the weeds about disk topologies and whether disk throughput makes any difference to the edit operation. The Rhino has decided to move all of the assets of a project to another machine to do the render there. He described a way to build a reasonably priced 10GbE network that allows him to save some of his life that he would have to spend waiting for files to move from one system to another.

Just two days ago, I decided to render half of a project on a machine in my garage and go on with the other half of the project on my main workstation, I sympathize with him.

While I can usually think of things to do for two minutes or so, a 10GbE would have saved me from the 20 minutes that it took just to get the files on to the remote machine over a GbE network. Who knows? 10GbE might allow one to work seamlessly on a remote machine using RDP. I've never had the pleasure of RDP over anything faster than GbE.

Then again, could you just have one copy of the source files if the storage server and the network were fast enough? All systems could map the storage server as the same disk and you could just move the project file from one machine to another with impunity. This is intellectual fodder to me, though I've seen more than my share of disk systems in my life.

 

fifonik wrote on 5/25/2019, 2:05 AM

I have not tried to say that 10G Ethernet useless (I'd like to have it as well as I am copying across my home network high volumes of data during backups. Not extremely important in my case as I'm not sitting and waiting while the background process finished). I also did not say that something wrong with the price (it is very good!).

However, OP posted technical details with obvious errors (wrong measurements, impossible read speed). Such information should not be kept without correction as someone may think the information is correct and made decision that would be just money loss For example, if someone already have RAID0/10 with some not recent HDDs on NAS/server and will invest in 10G Ethernet. This will change nothing as the storage will be limiting factor.

Camcorder: Panasonic X920

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 3700X, RAM: G'Skill 16 GB DDR4 3200, Graphics card: MSI RX470 4GB, SSD: Samsung 850 EVO SATA 250MB, HDDs: Seagate & Toshiba 2TB, OS: Windows 10 64-bit 1903

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17

TheRhino wrote on 5/26/2019, 3:58 PM

However, OP posted technical details with obvious errors (wrong measurements, impossible read speed).

My goal was to provide an affordable 10G networking solution, not to ague RAID speeds... My RAIDs are on hardware cards that have memory cache, some have 10,000 rpm drives, and I'm even setting-up a NAS with SSDs that serve as a cache... There are MANY online RAID calculators that allow you to plug-in YOUR hardware specs to determine anticipated speeds which will likely vary from mine...

10G networking especially helps when I am fulfilling expedited client work. In this business it's easy for minutes to turn into hours, especially if I make a mistake...

Workstation D with $1,350 USD of upgrades in April, 2019
--$500 9900K @ 5.0ghz
--$140 Corsair H150i liquid cooling with 360mm radiator (3 fans)
--$200 open box Asus Z390 WS (PLX chip manages 4/5 PCIe slots)
--$160 32GB of G.Skill DDR4 3000
--$350 refurbished, but like-new Radeon Vega 64 LQ (liquid cooled)

Renders Vegas11 "Red Car Test" (QSV) in 13s-14s when clocked at 5.0 ghz
(note: BOTH onboard Intel & Vega64 show utilization during QSV renders...)

Source Video1 = 4TB RAID0--(2) 2TB M.2 on motherboard in RAID0
Source Video2 = 4TB RAID0--(2) 2TB M.2 (1) via U.2 adapter & (1) on separate PCIe card
Target Video1 = 32TB RAID0--(4) 8TB SATA hot-swap drives on PCIe RAID10 card

10G Network using cheap Mellanox2 Adapters
Copy of Work Files, Source & Output Video, OS Images on QNAP 653b NAS
Blackmagic Decklink PCie card for capturing from tape, etc.
(2) internal BR Burners connected via USB 3.0 to SATA adapters
Old Cooler Master CM Stacker ATX case with (13) 5.25" front drive-bays holds & cools everything.

Workstations A, B & C are 6-core 4.0ghz Xeon 5660 or I7 980x on Asus P6T6 motherboards.

Musicvid wrote on 5/26/2019, 4:45 PM

Congratulations on a fine rig, which should meet or exceed your needs for years to come.

john_dennis wrote on 7/22/2019, 12:53 PM

Even though I've avoided system hardware upgrade and benchmarking threads like the plague since my next hardware upgrade is scheduled for Q1 2021, today, I'm 100% certain that my next forklift upgrade will be based on a 10GbE network. I might do it before the workstation upgrade if I grow out of my current archive sooner.

TheRhino wrote on 7/23/2019, 10:42 PM

What I like best about the 10G network is the ability to quickly backup both my source & target video files which typically consist of large intermediate files. Now that my network is fast, I have converted my RAID10 SATA arrays to RAID0 - because a backup always exists elsewhere on the network... This prevents drive speed from bottlenecking 4K editing...

Workstation D with $1,350 USD of upgrades in April, 2019
--$500 9900K @ 5.0ghz
--$140 Corsair H150i liquid cooling with 360mm radiator (3 fans)
--$200 open box Asus Z390 WS (PLX chip manages 4/5 PCIe slots)
--$160 32GB of G.Skill DDR4 3000
--$350 refurbished, but like-new Radeon Vega 64 LQ (liquid cooled)

Renders Vegas11 "Red Car Test" (QSV) in 13s-14s when clocked at 5.0 ghz
(note: BOTH onboard Intel & Vega64 show utilization during QSV renders...)

Source Video1 = 4TB RAID0--(2) 2TB M.2 on motherboard in RAID0
Source Video2 = 4TB RAID0--(2) 2TB M.2 (1) via U.2 adapter & (1) on separate PCIe card
Target Video1 = 32TB RAID0--(4) 8TB SATA hot-swap drives on PCIe RAID10 card

10G Network using cheap Mellanox2 Adapters
Copy of Work Files, Source & Output Video, OS Images on QNAP 653b NAS
Blackmagic Decklink PCie card for capturing from tape, etc.
(2) internal BR Burners connected via USB 3.0 to SATA adapters
Old Cooler Master CM Stacker ATX case with (13) 5.25" front drive-bays holds & cools everything.

Workstations A, B & C are 6-core 4.0ghz Xeon 5660 or I7 980x on Asus P6T6 motherboards.