0

I'm using beanstalkd to offload some work to other machines. The setup is a bit unusual, the server is on the internet (public ip) but the consumers are behind adsl lines on some peoples homes. So there is a linux server as client going out through a dynamic ip and connecting to the server to get a job. It's all PHP and I'm using pheanstalk library.

Everything runs smoothly for some time, but then the adsl changes the IP (every 24h hours the provider forces a disconnect-reconnect) the client just hangs, never to go out of "reserve".

I thought that putting a timeout on the reserve would help it, but it didn't. As it seems, the client issues a command and blocks, it never checks the timeout. It just issues a reserve-with-timeout (instead of a simple reserve) and it is the servers responsibility to return a TIME_OUT as the timeout occurs. The problem is, the connection is broken (but the TCP/IP doesn't know about that yet until any of the sides try to talk to the other side) and if the client blocked reading, it will never return.

The library seems to have support for some kind of timeouts locally (for example when trying to connect to server), but it does not seem to contemplate this scenario.

How could I detect the stale connection and force a reconnect? Is there some kind of keepalive on the protocol (and on the pheanstalk itself)?

Thanks!

Barmar
  • 741,623
  • 53
  • 500
  • 612
token47
  • 1
  • 3

2 Answers2

2

You could try to close each connection right after the request is answered and reopen a new connection each time.

There is no close() function but you deleting the Pheanstaly Object with unset($pheanstalk) will close it.

This explanation is quite helpful: Pheanstalk (PHP client for beanstalk) - how do connections work?

Community
  • 1
  • 1
rubo77
  • 19,527
  • 31
  • 134
  • 226
  • I don't see how that will help. You create a new connection, then call `reserve()` to wait for a job. If the server goes down, you'll wait forever. How is that any different from calling `reserve()` on a previously opened connection? The only case where this helps is if the server goes down right after returning a job, so your attempt to reopen fails. – Barmar Oct 25 '12 at 06:11
  • `reserve()` can have a timeout directly as an argument, so maybe you can set the `reserve()` function just for a certain time lets say 5 minutes and then stop reserving it and start a new `reserve()` call immediately? – rubo77 Oct 25 '12 at 11:50
  • If you do `reserve-with-timeout`, the timeout is implemented on the server, not the client, so if the server dies it will never time out. So our current plan is to set a client-side alarm to break out of `reserve` after a few minutes. We also need to do this if the connection to our MySQL server hangs. – Barmar Oct 25 '12 at 11:55
  • The solution with a VPN connection posted here in your google groups question seems the best to me: "making a vpn between the client and the server, so that the IPs of each end of the connection never change, even if the dynamic ip changes and the vpn reconnect" https://groups.google.com/forum/?fromgroups=#!topic/beanstalk-talk/eP_G1Ugsfb4 – rubo77 Oct 25 '12 at 12:01
0

I haven't tried it yet, but I came up with the idea of connecting to the beanstalk server through an SSH tunnel. We can enable the ServerAliveCountMax and ServerAliveInterval options on the tunnel, so that a network or server failure will cause the tunnel to close. This should then cause the pheanstalk client to report an error.

Barmar
  • 741,623
  • 53
  • 500
  • 612