6

I'm caching lots of data in memory to a local Redis server. The server will need to connect to hundreds of other servers in the same data center (as they all need to access the same information hosted by the Redis server).

In order to minimize latency and maximize throughput I plan on connecting all of the servers to the Redis server with a 40 Gigabit Ethernet (40 Gbit/s).

I understand how to connect a couple of servers to one server, but how would I go about connecting hundreds of servers to a single server? I mean, I don't see enough ports to connect so many servers...

Is there a device that will allow me to connect all of these servers together?

Stannaz
  • 3
  • 1
Levi Hibbitts
  • 85
  • 1
  • 3
  • Do you really need that bandwidth? You will have to create a star network just like with any other Ethernet setup. bond/trunk multiple ports for higher bandwidth and single link failure, consider multipath, as well as multiple synced redis instances. – NiKiZe Mar 22 '22 at 22:12
  • 43
    As kindly as possible : please hire a sysadmin for whatever you're doing. Hire a good one. Spend a lot of money on them. – mfinni Mar 23 '22 at 01:49
  • 4
    You have tested this design on plain old 1Gb ethernet and you found that Ethernet bandwidth was a problem, right? Or you calculated it, at least? – user253751 Mar 23 '22 at 13:49
  • 9
    Redis is apparently single-threaded. Can one thread push 40Gb of bandwidth? I really doubt it - you'll probably be slowed down by Redis. You could use more than one copy of Redis, but if you can use 5 copies of Redis on one server, you can also use one copy on 5 servers (that you already have) and save money by not buying the one big server just for Redis. – user253751 Mar 23 '22 at 13:50
  • 1
    Typically each rack has an Ethernet Switch, and each of these uplinks to a main Ethernet Switch. https://www.cisco.com/c/en/us/products/switches/what-is-an-ethernet-switch.html#~q-a – David Browne - Microsoft Mar 23 '22 at 14:52
  • 16
    It's pretty strange to encounter a programmer who's never heard of an Ethernet switch, but [you're one of today's lucky 10000](https://xkcd.com/1053/) – user253751 Mar 23 '22 at 15:57
  • 2
    Consider your redundancy setup too - one single redis server is a SPOF and if it is production-impacting then an outage will be bad. – Criggie Mar 23 '22 at 18:29
  • 5
    "Hundreds of servers in the data-center" - Is the data-center already setup? If yes, your server can just be connected to a switch in the existing network of the data-center. If not, then the question should be "How do I setup a data-center with 100s of servers". – RedBaron Mar 24 '22 at 07:43
  • 1
    So... machines fail. A lot. Have you thought what happens if your single cache server fails? Or needs to be taken down for maintenance? Perhaps some sort of Redis cluster would be a better fit? – thkala Mar 24 '22 at 19:04
  • If anyone is looking for a question that is getting an unjustified amount of positive attention, here it is. – marsh-wiggle Mar 25 '22 at 11:46
  • If these are all the same o/s, and one o/s is compromised, then all the others will be too in very short order. – Bib Mar 25 '22 at 12:59
  • If you truly don't understand or wasn't previously aware what a network switch does, I would recommend researching network hardware a lot more before investing on actual hardware. 40 Gbps connections are pretty expensive and you shoudn't get all that hardware just for fun. Even 2x10 Gbps in 802.3ad (LACP) configuration with redundant switches gets you plenty of bandwidth and the latency for each server often starts to be bigger bottleneck than the network bandwidth. And you don't lose the whole network when a single switch fails. When you have more than 3 servers, you should make it redundant. – Mikko Rantalainen Apr 08 '22 at 09:08

3 Answers3

14

I understand how to connect a couple of servers to one server but how would I go about connecting hundreds of servers to a single server? I mean, I don't see enough ports to connect so many servers.

Well... you would need to connect all of the servers to an appropriately sized network switch. They've been in use for decades. The first commercial multiport Ethernet network switch was introduced in 1989.

joeqwerty
  • 109,901
  • 6
  • 81
  • 172
9

I'm caching lots of data in memory to a local redis server. The server will need to connect to hundreds of other servers in the same data center (as they all need to access the same information hosted by the redis server).

The amount of data you store/cache is rather irrelevant. Relevant is the speed at which you need to transmit data to or from storage.

In order to minimize latency and maximize throughput I plan on connecting all of the servers to redis server with a 40 Gigabit Ethernet (40GbE).

For a large number of high-volume network ports you need a good infrastructure design. You not only require a huge bandwidth inside the redis server - network, storage and processing - but also the means to distribute that bandwidth.

Depending on the exact size, a large chassis switch (up to 800 ports or so) or a hierarchical tree is required. This paper from Cisco should provide a good starting point. A collapsed-core design is likely sufficient for your size.

Zac67
  • 10,320
  • 2
  • 12
  • 32
  • 8
    “hundreds of servers” talking to a single-threaded Redis instance don’t need a developer to buy a million dollar switch. They need to use that money to hire someone who has *heard* of a switch, and then buy a couple of 48-port unmanaged Gigabit switches. – Jacob Krall Mar 24 '22 at 13:40
  • @JacobKrall We don't know how well that Redis server scales or whether it is *single-threaded* - I was trying to point out the necessity to evaluate the *network, storage and processing* requirements. – Zac67 Mar 24 '22 at 14:30
2

Well, it depends on your needs.

You may ask your distributor where you want to buy that hardware. They want to help you to get that deal.

One example:

If the hundreds of servers have only gigabit you might use up to 10x HPE 2930M stacked with stacking modules and stacking cables. Then add a 40 Gbit/s module in at least two of them for redundancy and connect your server with both modules via DAC cables.

The 2930M has up to 48 ports, so 480 servers + the one 40G server. Depending in your needs select different switches.

If you don't need the flexibility, but more throughput you should go with a modular switch.

You can find help on several vendors sites, but here I give you HPE’s solution for finding a switch if you want to do it all alone:

HPE Networking Switch Selector

Again: ask someone who sells that solution to you if he/she can help with the order. Otherwise, you might get in trouble when trying to put everything together and you notice that there are different power supplies, modules, cables that fit into different models. Some 2930Ms, for example, need more power than other 2930M models.

Peter Mortensen
  • 2,318
  • 5
  • 23
  • 24
An-dir
  • 154
  • 5
  • 6
    Asking a salesperson for a complex technical solution is a recipe for a disaster. – fraxinus Mar 23 '22 at 15:45
  • For that size of course not a salesperson. The Distributors i worked with had technicans/architects to help in cases like that – An-dir Mar 23 '22 at 18:04
  • 1
    Asking anyone with a vested interest in upselling, whether that's a salesperson or the distributor's technical advisor, is still a recipe for disaster :-) – thkala Mar 24 '22 at 19:02