Workhorse for 10Gbps Internet

[safc10g]

Hi all

Wanted to pick your brains on the best kit to go with a new 10Gbps sequential FTTP install. So far, thought process is:

1) Buy a 10Gbps switch - seems plenty about, any suggestions?
2) Buy a workstation/server with a 10Gbps NIC in and some NVME drives to be able to cope with the throughout
3) Utilise my current WiFi6 mesh kit (ASUS) to feed around the house and existing gigabit switches for kit that doesn’t need 10Gbps (most!)

This is where I’m at - still can’t decide the best way to proceed!

Probably going to run something like proxymox on the machine, so I can virtualise different things (Plex, cctv, firewall, vpn etc..) so I’m trying to decide the best sort of machine would be best here.

1) Second hand workhorse, with dual Xeon and loads of ram:

HP Z820 WORKSTATION (24Cores/48Threads)

MAIN SPECIFICATION:

Processors: Dual, 2 x Intel Xeon E5-2697V2 (2.70/3.50GHz, 12C/24T)
Memory: 256GB DDR3 ECC 1866MHz
Graphics: NVidia Quadro K5200 (8GB GDDR5) GPU. (Probably sell this and buy a P2000 for better Plex transcodes)
Storage: 256GB SSD New (OS)
3 x Caddy Trays included for additional HDDs
Optical drive: DVDRW+/-
Power Supply: 1125W 90% High Efficient PSU
Integrated: Gigabit Ethernet
Integrated: HD Audio
Ports: 4 x USB 3.0, 5 x USB 2.0
OS: Windows 10 Pro

(£1600 for the above)


2) Newer machine - any suggestions based on the above (rough) use case?

 

[faceman123]

I don't know much but for £1600 I'd go threadripper or ryzen

 

[safc10g]

I don't know much but for £1600 I'd go threadripper or ryzen

I guess the issue with that would be, by the time I add 256GB ECC RAM, the price would go nuts? Want to get as much RAM in as possible for the VM’s - the more the better was my thought there!

AFAIK nothing that I plan to use it for will be hammering on the CPU’s - especially once I swap the K5200 out for a P2000, as that will then hardware transcode in Plex.

(The above Z820 also has a dual 10Gbps network card in, forgot to add that!)

 

[Avalon]

The whole question seems strange. We have literally ‘Plex, CCTV, Firewall, VPN etc.’ as intended usage and you’ve spent more time telling us about Plex transcodes (hint: Local/remote user numbers? Concurrent transcodes? Media format? Clients?) and the strange choice of a P2000 and an ancient and dubious £1600 workstation than any anticipated workloads. Have you looked at the issues of virtualising a firewall, let alone 10Gb WAN and doing anything more than basic NAT? Which firewall? Are you aware of the speed limitations on OpenVPN? Even Wireguard will struggle to get much over 5Gbit and that’s bare metal. Your costs in year one are going to be over 5K, that’s getting on for 12 years of decent dedicated server hosting in a DC. The RAM point is fair, but nothing you’ve suggested will get anywhere close to needing 64GB, let alone 256GB, what aren’t you telling us? Those CPU’s are power hungry and inefficient by today’s standards (I killed off my v3/v4 kit this year), a 3900x comfortably beats them in CPU Mark (14Kx2 vs 32K in under half the power), a 3950x manages 39k.

For comparison I pay under £40/m for a decent EPYC shared 10Gb set-up with more RAM/cores than I need, that lets me pull 5Gbit+ whenever I want and if I want another one, it’s only a few clicks. You’re paying £225/m and buying old hardware for £1600 with likely another grand on top for anything approaching a reasonable storage/GPU/10Gb switch/card set-up, year one is over £5K easily, or about 12 years of hosting on a dedicated box with unlimited symmetrical gigabit or shared VPS with 10Gbit.

Please do a lot more reading/thinking/explaining before going any further, start with what you want to do and don’t fall into the obvious trap of ‘I need ancient enterprise class hardware, because enterprise!’.

 

[safc10g]

The whole question seems strange. We have literally ‘Plex, CCTV, Firewall, VPN etc.’ as intended usage and you’ve spent more time telling us about Plex transcodes (hint: Local/remote user numbers? Concurrent transcodes? Media format? Clients?) and the strange choice of a P2000 and an ancient and dubious £1600 workstation than any anticipated workloads. Have you looked at the issues of virtualising a firewall, let alone 10Gb WAN and doing anything more than basic NAT? Which firewall? Are you aware of the speed limitations on OpenVPN? Even Wireguard will struggle to get much over 5Gbit and that’s bare metal. Your costs in year one are going to be over 5K, that’s getting on for 12 years of decent dedicated server hosting in a DC. The RAM point is fair, but nothing you’ve suggested will get anywhere close to needing 64GB, let alone 256GB, what aren’t you telling us? Those CPU’s are power hungry and inefficient by today’s standards (I killed off my v3/v4 kit this year), a 3900x comfortably beats them in CPU Mark (14Kx2 vs 32K in under half the power), a 3950x manages 39k.

For comparison I pay under £40/m for a decent EPYC shared 10Gb set-up with more RAM/cores than I need, that lets me pull 5Gbit+ whenever I want and if I want another one, it’s only a few clicks. You’re paying £225/m and buying old hardware for £1600 with likely another grand on top for anything approaching a reasonable storage/GPU/10Gb switch/card set-up, year one is over £5K easily, or about 12 years of hosting on a dedicated box with unlimited symmetrical gigabit or shared VPS with 10Gbit.

Please do a lot more reading/thinking/explaining before going any further, start with what you want to do and don’t fall into the obvious trap of ‘I need ancient enterprise class hardware, because enterprise!’.

All very much just bouncing ideas around at the min, nothing set in stone...hence no firm requirements yet. Was just seeing what peoples thoughts were and then would expand from there...

 

[Avalon]

If nothing is set in stone and you have no clear idea of what you intend to use the hardware for, or why you want/need/how you will use 10Gb WAN, why have you ordered it? If it’s for the internet points and money is no object, you wouldn’t be looking at ancient over priced Xeon workstations. Intended usage generally dictates hardware specs, or is this just all hypothetical, in which case we’ve had similar threads in the past.

Which provider is it? Do they supply hardware and if so what? Presumably you get multiple IP’s? Are you just doing NAT or full UTM and which specific district are you intending to virtualise? VPN wise if you are running via a remote end point have you considered that very few (eg not the race to the bottom brigade) use anything other than cheap rented gigabit boxes which are awful for a variety of reasons, a few in some locations use 10Gb and fewer still own them, but it’s shared over a *lot* of users. Which NIC’s and are you sure they play nicely with Proxmox? Did Proxmox offer something specific that ESXi didn’t? What connectivity standard are you going for as this dictates card/switch type? Why 256GB when nothing you have posted suggests you will get past 16GB? Plex won’t be using your RAM extensively as you aren’t transcoding to it (that P2000 is going to be doing it and it’s RAM is your limit), CCTV wise what software are we talking about? Number of cameras, resolution, frame rate and if you plan on re-encoding before writing? As you know AHCI SSD’s crap out at 550MB/s, so as per your previous post you need large amounts of fast NAND and if it’s going to be used as intended, it needs endurance. I like the fusion.io stuff but drivers need consideration, intel's P3xx0 tend to be much more £/GB, but decent on endurance depending on the model, it all comes down to what sort of workloads are you planning.

 

[safc10g]

If nothing is set in stone and you have no clear idea of what you intend to use the hardware for, or why you want/need/how you will use 10Gb WAN, why have you ordered it? If it’s for the internet points and money is no object, you wouldn’t be looking at ancient over priced Xeon workstations. Intended usage generally dictates hardware specs, or is this just all hypothetical, in which case we’ve had similar threads in the past.

Which provider is it? Do they supply hardware and if so what? Presumably you get multiple IP’s? Are you just doing NAT or full UTM and which specific district are you intending to virtualise? VPN wise if you are running via a remote end point have you considered that very few (eg not the race to the bottom brigade) use anything other than cheap rented gigabit boxes which are awful for a variety of reasons, a few in some locations use 10Gb and fewer still own them, but it’s shared over a *lot* of users. Which NIC’s and are you sure they play nicely with Proxmox? Did Proxmox offer something specific that ESXi didn’t? What connectivity standard are you going for as this dictates card/switch type? Why 256GB when nothing you have posted suggests you will get past 16GB? Plex won’t be using your RAM extensively as you aren’t transcoding to it (that P2000 is going to be doing it and it’s RAM is your limit), CCTV wise what software are we talking about? Number of cameras, resolution, frame rate and if you plan on re-encoding before writing? As you know AHCI SSD’s crap out at 550MB/s, so as per your previous post you need large amounts of fast NAND and if it’s going to be used as intended, it needs endurance. I like the fusion.io stuff but drivers need consideration, intel's P3xx0 tend to be much more £/GB, but decent on endurance depending on the model, it all comes down to what sort of workloads are you planning.

Hey, no other reason than I have had rubbish broadband/FTTP for as long as i can remember. I always wanted more...my trusty old 56k modem was never enough, so once I discovered EOL (Europeonline) offered Satellite Internet, straight in there! DVB-S card installed and dish pointed to 19.2e and I was set. Yes it was rubbish, but you get my point

Provider for the connection is a new startup called YouFibre - ran by the same guy who owns/runs Community Fibre I believe.

I still have the option to change back to 1Gbps (£30/month and have up to 3 more 1Gbps connections at £27/month each - so this could probably work out a much better option....let’s see...)

Not sure if I will get multiple IP’s - one static I think, but will find out more. Really not thought much about the firewall side of things, need to investigate more here. They provide only the ONT, no other kit for the business service. They provide Eero Pro’s for the consumer service. I’m currently thinking just going with a switch that has a few 10Gbps ports will suffice, I can then just connect to my own routers.

The Xeon workstation was just something I had saw whilst searching around, I get your point now (thanks for the pointers) that it’s useless. So wont progress with that.

To try and clarify more on intended usage would be for the new machine and the connection:

• CCTV
  • Probably Blue Iris as the software
  • 3 x Hikvision 4K cameras - recording at 4K/30fps - currently connected to a Hikvision NVR with a 6TB WD Red Drive inside. No re-encoding, they will record direct to H.264 or H.256. this works well currently on the quite low powered NVR.
  • 1 x Ring Floodlight Cam (Not sure if this will work - not the end of the world if it has to remain within the ring ecosystem)
  • 1 x Ring video doorbell (As above)
• Plex server
  • Mixture of 4K HDR and 1080p - MKV Remux mainly to retain quality
  • Local and remote - only a handful of family members and friends will ever connect.
    • This was where the transcoding thought process come into it - but on second thoughts, I will just disable 4K for anyone who either doesn’t have a 4K TV or their Internet is not fast enough to direct play.
  • I have my media stored on a WD EX4100 NAS (56TB - 4 x 14TB WD RED drives)
  • The media is also mirrored onto google drive for redundancy
  • Maybe the i9-9900K as a base (with the iGPU being able to hardware transcode for plex which is an added bonus if ever needed before I even look at dGPU’s?)
  • RAM wise - probably was being ridiculous with the 256GB - 64 or 128 seem very reasonably priced
• Firewall
  • really have not thought much about this - so this is TBC
• VPN
  • As above - good call on the speeds, as dont want to hinder the connection with rubbish servers. I must admit I thought you could just run it off your own and utilise your own bandwidth. Used to have similar on a Hetzner dedicated server (granted it was 1Gbps dedicated and not 10, so maybe becomes an issue above 1?)
• Storage
  • As mentioned above will store files on local NAS - also mirror up to google drive for redundancy
  • Will look to have at least 2 x NVME in the new machine to avoid the storage being a bottleneck for any download/upload
• Gaming
  • I’ve had a change of plan now and will use the machine for gaming (not frequently, maybe VR too with my Oculus Quest headset)
  • Looks like the new Nvidia 3080 is going to be the same price as the 2080 currently is - so might get one of these when they are released (later this month I believe?)

That’s probably it for now!

 

[HEADRAT]

For comparison I pay under £40/m for a decent EPYC shared 10Gb set-up with more RAM/cores than I need, that lets me pull 5Gbit+ whenever I want

Can you provide some details, sounds great

I still have the option to change back to 1Gbps (£30/month and have up to 3 more 1Gbps connections at £27/month each - so this could probably work out a much better option....let’s see...)

This sound like a much more sensible option, is the line symmetric or asymmetric?

 

[Bluecube]

OP - I’d keep the gaming separate from the server functions especially if you’re planning on recording CCTV. CCTV can and does hit the CPU/ storage functions very hard and will affect gaming performance.

Purely a personal preference but I would be looking to hive out the different functions to different hardware eg a server for Plex and CCTV, some sort of high end router for the WAN and firewalls, and a gaming PC for gaming. I think that if you combine everything you want into a single PC then it’ll be a bit of a car crash...

 

[safc10g]

This sound like a much more sensible option, is the line symmetric or asymmetric?

Symmetric, 10Gbps up and down. That was what made me jump at it! Haha. So its 1x10Gbps for £225 or 4 x 1Gbps for £111

I know which one I should go for and which is the sensible one....but then I know I will probably stick with the 10 and then I can add to what I use it for as I go

OP - I’d keep the gaming separate from the server functions especially if you’re planning on recording CCTV. CCTV can and does hit the CPU/ storage functions very hard and will affect gaming performance.

Purely a personal preference but I would be looking to hive out the different functions to different hardware eg a server for Plex and CCTV, some sort of high end router for the WAN and firewalls, and a gaming PC for gaming. I think that if you combine everything you want into a single PC then it’ll be a bit of a car crash...

Good points above, probably (as always!) looking to do too much at once. I may just leave the CCTV running on the NVR, I mean that was what I bought it for at the end of the day, so probably no need to even touch that! Just means I can give it plenty of bandwidth to upload from the NVR to the cloud I guess. Doesn’t need to touch the new machine.

Plex shouldn’t impact too much I guess, as there’s only ever going to be a handful of people on at any point in time. As the iGPU for something like the 9900K can handle the odd transcode as well that would leave me plenty, I assume?

Reading all of the above points (thanks for them!) I am realising very quickly that I am just trying to do things I don’t even need lol!

May go with a lesser powered PC and then a more expensive dedicated firewall/switch - keep things to do the jobs they were designed for.

 

[Bluecube]

If you go down the separate server route will you even need such a powerful CPU? Plex transcoding can be avoided by choosing appropriate file formats for your streaming devices and if it’s needed then an i5 would be more than enough.

I think this is more a networking question in how to receive and distribute such a fast broadband so I’d focus more on the router side of things and see if you can hardwire cables across the house. 10Gb network cards, whilst not cheap aren’t as pricey as they once were so build the PC/ server you want and then pop in the cards.

 

[safc10g]

If you go down the separate server route will you even need such a powerful CPU? Plex transcoding can be avoided by choosing appropriate file formats for your streaming devices and if it’s needed then an i5 would be more than enough.

I think this is more a networking question in how to receive and distribute such a fast broadband so I’d focus more on the router side of things and see if you can hardwire cables across the house. 10Gb network cards, whilst not cheap aren’t as pricey as they once were so build the PC/ server you want and then pop in the cards.

Sound advice. The spend is becoming less and less the more I think about it and read the useful comments here!

House is cabled with Cat7, so that’s nice and future proofed thankfully.

Would you suggest I go down the switch/router combined route, or just a switch and use my current WiFi 6 routers?

 

[Bluecube]

I’ve not a clue now! @Avalon may be able to help you. I’ve a suspicion you’ll need an enterprise level router to deal with speed but don’t quote me on that.

 

[safc10g]

I’ve not a clue now! @Avalon may be able to help you. I’ve a suspicion you’ll need an enterprise level router to deal with speed but don’t quote me on that.

Will keep this thread to a PC build I guess as venturing off topic again. Will pop a thread into the networking section re the switch/router.

Thanks so far for all the replies, certainly food for thought and helped

 

[Bluecube]

For your server usage, I'd look at standard PC equipment. My server (which runs Unraid) for example is a Ryzen 2700 on a Gigabyte Aorus motherboard with 32Gb of 2400Mhz RAM. It's used for Plex, some VMs, a few dockers and general storage. A Ryzen 2700 is overkill for my current usage requirements. You could take this as a benchmark and work out your own requirements from there. I doubt you'll need anywhere near as powerful a server as you think you do. You may even be happy with a NAS fitted with a 10Gb network card.

 

[safc10g]

For your server usage, I'd look at standard PC equipment. My server (which runs Unraid) for example is a Ryzen 2700 on a Gigabyte Aorus motherboard with 32Gb of 2400Mhz RAM. It's used for Plex, some VMs, a few dockers and general storage. A Ryzen 2700 is overkill for my current usage requirements. You could take this as a benchmark and work out your own requirements from there. I doubt you'll need anywhere near as powerful a server as you think you do. You may even be happy with a NAS fitted with a 10Gb network card.

Thanks for that, food for thought!

Still bouncing around ideas for networking kit to handle the 10Gbps, so may yet end up maxing out a machine to act as a router/firewall as well. No idea yet, so much to consider haha!

 

[WJA96]

Which Hikvision cameras do you have that record 30fps at 4K?

 

[safc10g]

Which Hikvision cameras do you have that record 30fps at 4K?

Sorry that’s a typo, 20fps, not 30. Cameras are DS-2CD2385G1-I

 

[safc10g]

Update.

Ended up going with a bit of a hybrid to cover all use cases, both present and future!

Unraid is the OS of choice.

1x i9-10940X
128GB RAM
2 x 2TB NVME (Other VM’s and/or fast data storage - which will then be moved off to for archive/uploaded to google drive)
1 x 500GB SSD (Windows VM)
1 x NVIDIA 2080ti (Gaming, of course)
1 x NVIDIA 1650 (Dedicated to plex transcoding)
4 x 14TB WD Red drives, going to take them out of the WDEX4100 enclosure and run direct in Unraid, as the WD is very limited anyway (only got it as was cheaper than buying the storage separately as got a very nice discount!)
1 x Gigabit LAN
1 x 10G LAN
1 x Zyxel AX7501-B0 Router (dual 10G ports, 1 RJ45 and 1 SFP+

Been playing with unraid for a few days, its driving me mental but I’m getting there!

May also bypass the NVR and plug the cameras straight into a switch in the loft and put the cameras HDD (6TB WD RED) into the array as well and just have one machine powered on instead of multiple.

Let the “fun” begin!