5

I'm trying to build instant messaging app. Clients will not only send messages but also often send audios. And I've decided to use websocket connection to communicate with clients. It is fast and allows to send binary data.

The main idea is to receive from client1 message and notify about it client2. But here's the thing. My app will be running on GAE. And what if client1's socket is opened on server1 and client2's is opened on server2. This servers don't know about each others clients.

I have one idea how to solve it, but I am sure it is shitty way. I am going to use some sort of communication between servers(for example JMS or open another websocket connection between servers, doesn't matter right now).
But it surely will lead to a disaster. I can't even imagine how often those servers will speak to each other. For each message server1 should notify server2, server2 should notify client2. But things become even worse when serverN comes into play.

Another way I see this to work is Firebase. But it restricts message size to 4KB. So I can't send audios via it. As a solution I can notify client about new audio and he goes to my server for it.

Hope I clearly explained the problem. Does anyone know how to solve it? Or maybe there are another ways to build such apps?

Vladislav Gutov
  • 106
  • 1
  • 6
  • question is somewhat broad. each part has its own solution. for example for audio send the id of the saved audio, not the audio data itself. – Zig Mandel Jul 13 '16 at 13:36
  • yes, I have been thinking to do it like that, but still. The main problem is somehow notify another server about client1's message to client2 – Vladislav Gutov Jul 14 '16 at 09:07
  • sorry my comment got cut off. Its not an issue if your servlet is stateless. Store everything in datastore (with caching) and the message contains all the needed data to identify it in datastore. Also consider firebase. – Zig Mandel Jul 14 '16 at 17:51
  • Basically, you offer me save message in database(or cache) and then retrieve it on the other side? If yes, won't it be too heavy for server and database? Let's say that each client polls database every second. They will overload the database. Or am I wrong? Friend of mine found that akka has solution for actors' communication in distributed system. It allows to search them by name as if they were created in a single machine. I haven't checked that info yet, but I hope it does. Maybe I should use them then. – Vladislav Gutov Jul 14 '16 at 19:04

2 Answers2

3

If you are building a messaging cluster and expect communicating clients to connect to different instances of the server then server-server communication is inevitable. Usually it's not a problem though.

  • First, if you don't use any load balancing your clients will connect to the same server 50% of time on average (in case of 2 servers).
  • Second, intra-datacenter links are fast and free in all known public clouds.
  • Third, you can often do something smart on the frontend to make sure two likely to communicate clients connect to the same server. For instance direct all clients from the same country to the same server using DNS load balancing.

The second part of the question is about passing large media files. It's a common best practice to send it out of band - store on the server and only pass the reference to it. Like someone suggested in the comment, save the audio on the server and just send a message like "audio is available, fetch it from here ...". You don't need to poll the server for that. Just fetch it once when the receiving client requests it.

In general, it seems like you are trying to reinvent the wheel. Just use something off the shelf.

Gene S
  • 495
  • 5
  • 14
1
  1. Let all client get connected to multiple servers and each server keeps this metadata
  2. A centralized system like zookeeper stores active servers details
  3. When a client c1 sends a message to client c2:
  • the message is received by a server (say s1, we can add a load balancer to distribute incoming requests)
  • s1 will broadcast this information to all other servers to get which server the client c2 is connected to OR a better approach to use consistent hashing which decides which server the client can connect to & in this approach message broadcast is not required
  • the corresponding server responses to server s1 (say s2)
  • now s1 sends the message m to s2 and server s2 to client c2

Cons of the above approach:

  1. Each server will have a connection with the n-1 servers, creating a mesh topology

  2. Centralized system (zookeeper) becomes a single point of failures (which is solvable)

Apps like Whatsapp, G-Talk uses XMPP and TCP/IP.

Andrii Abramov
  • 10,019
  • 9
  • 74
  • 96
Bishnu
  • 383
  • 4
  • 14