1

We are trying to transfer large number of files from a network share into Openstack using a custom written Erlang application (using Erlang version R15B).

After about 20k files transferred the Erlang process starts returning enobufs errors resulting in it being un-able to establish HTTP connections to OpenStack. As well, C# code running on the same box can no longer open sockets resulting in errors seen in this question about lacking sufficient buffer space. Closing all the custom applications and restarting does not alleviate the situation.

The number of open connections never exceeds 300 connected but does typically go into the 2k range when including TIME_WAIT. Connections are not being held open but are being opened and closed quickly.

All the articles I could find seem to mention that 'ephemeral' port limit but it doesn't seem like we are reach that, and closing the Erlang application doesn't result in any improvements.

Is there something else that could be holding on to buffer space and not releasing it properly?

Community
  • 1
Eric
  • 31
  • 5
  • It does seem like you are reaching the ephemeral port limit. Why do you think it doesn't seem like you are reaching that? By default, your OS allows 16,383 ephemeral ports. If you wait a few minutes, does the problem go away? Does each file use its own TCP connection? How quickly do the files transfer? – David Schwartz Jul 13 '12 at 00:00
  • I think that I'm not hitting the limit because we are maxing out at close to 2-3k which is not near 16k. No, waiting any amount of time doesn't help and restarting the application doesn't help either. Error appear instantly on restart. – Eric Jul 13 '12 at 01:57
  • You said, "after about 20k files transferred". Each file is a connection. So you're past 16K. However, that restarting the application or waiting doesn't help does point away from ephemeral ports being the issue. – David Schwartz Jul 13 '12 at 02:33

1 Answers1

2

One of the functions of the application was to talk to a local service. It appears as if this triggered a kernel socket leak in Server 2008 where it would not release sockets properly. Applying the hotfix mentioned in the support article resolved the issue.

Eric
  • 31
  • 5