2

I have been thinking a lot about scaling out the WAMP router onto multiple machines. Especially because relying on one system without having a backup-system seems a bit tough in a live scenario.

To not have a complicated master node election and all that comes with it, I had the following idea and I would love to get some feedback.

Routers would need to share some information, i.e.:

Authentication

  • session ids

RPC

  • check if a client has already registered for an uri when using simple registration
  • forward calls for pattern based registration
  • meta api information

PUB/SUB

  • provide event messages to all clients in the network
  • meta api information

There must be a connection between all routers. To keep the connections and traffic low, the Idea is to have a ring infrastructure:

   +----------------+    +-----------------+    +--------------+
   |                |    |                 |    |              |
+-->  First Router  +---->  Second Router  +---->  Nth Router  +--+
|  |                |    |                 |    |              |  |
|  +----------------+    +-----------------+    +--------------+  |
|                                                                 |
+-----------------------------------------------------------------+

Every router would have it's own client that is connected to the next router. By storing the incoming message id for a CALL or PUBLISH that came from a client the router could identify a round trip. Incoming calls coming from another router would be handled as is. Yields would have the correct INVOCATION.Request|id and could be forwarded to the correct router.

LOOSING A ROUTER

This infrastructure forces to have some static and identical configuration for all routers. This means dynamic scaling would not work, but therefor no leader election is necessary. If a routers is killed the next router will be connected from the list of routers. A constant polling could check for the router to be back online.

I have not done any deep research on this. So I would be happy to get some input for that.

Community
  • 1
  • 1
Richard Burkhardt
  • 781
  • 7
  • 25
  • "This means dynamic scaling would not work" - but why we can't just keep routers IP list shared between routers? "Routers would need to share some information, i.e. session ids" - but what is the reason to share this information if routers can just proxify requests between each other? – barbushin Feb 15 '16 at 09:55
  • How would you extend the list of routers dynamically? If it is a configure file you would need to poll it. If it is somehow generated you would need leader election to have someone to provide the information to the rest. Another possibility is giving a port range that is to be sniffed. – Richard Burkhardt Feb 15 '16 at 10:12
  • New router connects to any of routers in cluster, and receive list of active routers IP's. So list of active routers is not in config file, it's just in memory of every single active router. FYI, that's how Bitcoin daemons works. – barbushin Feb 15 '16 at 14:54
  • How does the new router know where to connect to the first time before the the list of routers has been passed? – Richard Burkhardt Feb 15 '16 at 14:57
  • In Bitcoin there is a list(10-20) of master-nodes IP's configured(hardcoded) in all daemons & clients. Usually this nodes are in different Availability Zones, and to prevent highload they're used only for connection initialization and retrieving complete list of nodes. So for 10 routers cluster, you can just pick 2-3 routers to be a master-nodes, and configure all another routers and clients to use this routers for connections initialization. – barbushin Feb 16 '16 at 01:31
  • The wamp router does not need a very heavy scaling. The crossbar router can pass 30000 messages per second on a single node. The first use case for scaling would be reliability. There might be a list of 3 routers all up. This just pick 3 to be the master sounds like a lazy and dirty way of master node election. Mature systems like zookeeper have a proper implementation. But the implementation is very tricky. But yes, your way would probably work. – Richard Burkhardt Feb 16 '16 at 06:23

1 Answers1

0

nginx can work as a reverse proxy to accomplish this.

ip_hash on the upstream list will make sure a client IP get's the same server every time. You may or may not need this depending on your situation.

Essentially, you want a "Load Balancer" to stand in front of WAMP servers, and delegate connections as needed.

Michael Cole
  • 15,473
  • 7
  • 79
  • 96
  • nginx is no good solution for this. nginx does not know about any states and is not capable of sharing or restoring them. The wamp router is a proxy itself and does pretty much the same thing as nginx. – Richard Burkhardt Dec 08 '21 at 05:27
  • @RichardBurkhardt that may be true, but I wouldn't expose anything to the internet unless it was behind a proxy with proven security like nginx/apache/etc. If you don't understand why, you're playing with fire. – Michael Cole Dec 10 '21 at 22:06
  • Thats a good point, but I rely on netty whichshould have a proven security concept too. Do you have any source to your statement? Sounds a bit like it is impossible for me to write secure code on my own? – Richard Burkhardt Jan 11 '22 at 19:50
  • @RichardBurkhardt - essentially yes, unless you fuzz it. https://github.com/uNetworking/uWebSockets/tree/master/fuzzing – Michael Cole Jan 11 '22 at 23:39