1

Is it possible to scale the number of inbound TCP connections into Nifi to tens of thousands ? Scaling of NIFI with TCP connections : The docs state a max setting of 2.

We are expecting to handle between 10-25,000 long running TCP connections (max connection duration would be 4 hours). Deploying multiple redundant NIFI clusters to handle the load would not be a problem.

The domain is IoT with TCP devices. Devices can only send messages over TCP. Each device sends a message every 2 minutes. We are considering moving over to a NIFI cluster/containerised solution on AWS if it can scale to handle our connection load.

Any similar challenges or experiences or workarounds?

Thanks!

PerryS
  • 11
  • 1
  • How your devices works? Are they opening connection for a long time without closing it? – daggett Nov 18 '19 at 13:09
  • Generally Yes. Connection times vary- could be 30min 60 min 90min,etc... But the max length a device will stay connected is 4hrs. Then it is disconnected. This is a hard rule. Once disconnected a device will wait 20min and then go through a process to re-connect. – PerryS Nov 18 '19 at 13:23

1 Answers1

0

You can increase the max connections to whatever your NiFi instances can handle. There will be a thread reading from each connection, so it would end up creating 10k threads if there were 10k concurrent connections to a single instance. You would likely want a TCP load balancer in front of a NiFi cluster, so maybe a 10 node cluster would get you 1k connections per nod.

Bryan Bende
  • 18,320
  • 1
  • 28
  • 39
  • Thanks ! Regarding the 10 node cluster, is it feasible to deploy it based on a cloud containerised infra service like AWS (Elastic Container Service) or AWS EKS (Elastic K8 Service) ? – PerryS Nov 19 '19 at 10:30