2

I'm running a multithreaded daemon where an instance of ruby Mechanize (which contains a Net::HTTP::Persistent object), might be used and run by one of many threads. I'm running into tons of problems because Net::HTTP::Persistent opens a new connection for each thread that runs it, so if I have 50 threads, i end up opening 50 times more connections than i need to! I've tried subclassing and patching Net::HTTP::Persistent to store its connection information as part of its class instead of in Thread.current, but then I keep getting 

too many connection resets (due to Broken pipe - Errno::EPIPE)

all over the place.. any thoughts? anyone know an alternate library to Net::HTTP::Persistant I could use, and hopefully easily patch Mechanize with?

Mohamed Hafez
  • 8,621
  • 7
  • 41
  • 49
  • It seems to me that you are assuming mechanize is thread safe, but that's not a good assumption to make. Is this MRI or JRuby? – pguardiario Aug 29 '12 at 00:42
  • Its MRI, 1.9.3. I make sure that only one thread can be using a given agent at a time (using mutexes) – Mohamed Hafez Aug 29 '12 at 01:10
  • 1
    Have you tried mechanize 1.0.0 (which doesn't use persistent connections)? – pguardiario Aug 29 '12 at 01:35
  • Yeah i've been trying that and its been stable so far, even with tons of threads. I know it doesnt use Net::HTTP::Persistent, but does it really not use Persistent connections? When i'm in IRB and get() a page, it i can see the port open and say ESTABLISHED, and if I get() again it seems to reuse the same port, plus it does set keep-alive and all that. Either way i'm just happy the things running again, just curious – Mohamed Hafez Aug 29 '12 at 04:49
  • 2
    I think there's confusion about what a persistent connection is. It has less to do with the connection object being reused and more to do with skipping the initial part of the http handshake once you've already done so with a remote server. – pguardiario Aug 29 '12 at 06:10
  • Nope, looks like Mechanize 1.0 is reusing the same TCP connection and fully implementing a persistent connection. I took a look at the code and then verified with Wireshark. It just does it in a reliable way thats friendly to multithreading, unlike Net::HTTP::Persistent – Mohamed Hafez Aug 29 '12 at 19:48

2 Answers2

1

The problem is, if you access a Net::HTTP::Persistent object from another thread, and that object is in the middle of something, that thread would either have to block (stop execution and wait for the object to do what it needs to), or create a new object and mess with that. With threading, you could be in the (forgive me, I'm making assumptions here) middle of a HTTP request, when all of a sudden, another thread wants to create a HTTP request using the same connection, which breaks everything (probably why you got the connection reset issue).

If you really want threading, your options are to have however many connections open, or wait for an open connection so you can use it.

Jeremy Rodi
  • 2,485
  • 2
  • 21
  • 40
1

Reverted back to Mechanize 1.0.0, and that solved the problem. Persitant connections are handled in a more reliable, multithreading friendly way in 1.0, unlike in Net::HTTP::Persistent which Mechanize 2+ uses. My advice: stick with Mechanize 1.0 its more reliable, gets less errors, and DOESNT HAVE CRAZY PROBLEMS WITH MULTITHREADED CODE!!! sheesh.

Note: Unlike some of the comments may suggest, Mechanize 1.0 DOES implement persistent connections: take a look at the source code, or verify with Wireshark.

Mohamed Hafez
  • 8,621
  • 7
  • 41
  • 49
  • Again, they may be persistent in the sense that they persist, but not in the sense that the rest of us mean when we talk about persistent connections. – pguardiario Aug 29 '12 at 21:52
  • 1
    How about persistent in the sense that they don't redo the TCP handshake or SSL verifications, which seems to be wikipedia's definition: http://en.wikipedia.org/wiki/HTTP_persistent_connection – Mohamed Hafez Aug 30 '12 at 01:10