4

I'm working on JPMorgan's Quorum where I wrote a simplistic private smart contract and I'm hitting it with as much as possible of transactions to stress test it. The configuration I'm using is the 7nodes example. (https://github.com/jpmorganchase/quorum-examples)

I ended-up with 200 Tx/sec. When I checked the logs I found that constellation-node is logging "Too Many Open Files" while trying to connect to other constellation nodes using Unix IPC.

I raised the soft and the hard limits of file descriptor to 1 million with no use as it seems that constellation-node is not honoring the limit. By listing the file descriptors under /proc//fd I'm always getting less than 1024 files.

I'm not sure what I'm doing wrong here and I hope that someone can help.

Cheers!

M. K. Q.
  • 51
  • 5
  • Did you change the ulimit settings too? – NopOp Dec 19 '17 at 15:45
  • I changed the file descriptor limits both system and user-wide – M. K. Q. Dec 19 '17 at 16:14
  • Hm, and you've restarted the machine after those changes, set both hard and soft limits for both fd and ulimits, aren't running the service via systemd or screen? If you're ssh'ing in you might also need to add UsePAM to sshd-config. – NopOp Dec 19 '17 at 16:38
  • Already did all that and yes, I'm using ssh :| The maximum number of unix sockets for 7 processes is not exceeding 6K and I'm still getting "Too Many Open Files" This is the error I'm getting: Post http+unix://c/send: dial unix qdata/tm2.ipc: socket: too many open files – M. K. Q. Dec 19 '17 at 16:45
  • The errors received on the client-side were misleading (found in the original post). The actual error message logged on the server-side is: http: Accept error: accept tcp [::]:22000: accept4: too many open files; retrying in 5s. – M. K. Q. Dec 20 '17 at 15:58
  • The culprit is in Quorum itself @ https://github.com/jpmorganchase/quorum/blob/0905eda48eb07a4ce0e7072c1a2ecbf690ddff77/private/constellation/node.go#L86 The function "do" is opening connections and deferring there closure, also it's not setting the HTTP "Connection" header to "Close". – M. K. Q. Dec 21 '17 at 08:08

0 Answers0