1

I'm looking for a streaming server and the docs on NATS Streaming don't make it clear what scaling strategy will be possible with it in my case.

I assume we'll be using FT mode since we need to ensure best message durability, but that means the whole set of channels will be handled by only one server at a time.

It is also possible to partition channels, thus allowing multiple server FT groups to coexist in a single mesh and split the set of channels between different servers.

What is not clear however is the following.

If I chose the application model where every single entity receives a dedicated unique channel, and I can have millions of coexisting entities (e.g. active customers) - will NATS be able to handle millions of channels at once, in FT+Partitioning mode?

There likely will be only one publisher and consumer per channel in this case.

alexykot
  • 699
  • 1
  • 8
  • 21

1 Answers1

0

Having millions of active consumers (corresponding to unique channels) is a stretch that I am not sure NATS Streaming or the system it is running on will be able to handle. Note that a channel is represented on disk by a directory and that directory then contains files for messages (data + index) and for the subscriptions state on this channel. So you may run into file descriptor issues.

Not clear from your question if the consumers are 1 for each channel, but if not and you possibly have a high number of consumers for the same channel, then you need to think also about fan-out issues where a message has to be delivered (TCP sent) to that many consumers.

I. Kozlovic
  • 796
  • 3
  • 3
  • It's likely will be one consumer per channel, clarified that in the question. Also, I'm expecting to back it by SQL, so file descriptors should not be a problem. Or will some tmp files be created even if the state is in SQL? – alexykot Nov 18 '20 at 16:28