0

While upgrading a large project from 1.8.7 to 1.9.2, I have uncovered a bug that I am at a loss to explain. It appears the fork operation breaks a memcached connection, even when the memcached connection isn't used in the forked code. Here is an IRB session, in 1.9.2 showing the error, and in 1.8.7 showing no error. Can anyone explain why there is no ServerIsMarkedDead error in 1.8.7 but there is in 1.9.2?

MBPR:$ rvm list

rvm rubies

   ruby-1.8.7-p370 [ i686 ]
=* ruby-1.9.2-p320 [ x86_64 ]

# => - current
# =* - current && default
#  * - default

MBPR:$ irb
irb(main):001:0>     require 'rubygems'
=> false
irb(main):002:0>     require 'memcached'
=> true
irb(main):003:0>     mc = Memcached.new("localhost:11211", :timeout => 1.0)
=> #<Memcached:0x007fc91082abc8 @struct=#<Rlibmemcached::MemcachedSt:0x007fc91082a948>, @options={:hash=>:fnv1_32, :no_block=>false, :noreply=>false, :distribution=>:consistent_ketama, :ketama_weighted=>true, :buffer_requests=>false, :cache_lookups=>true, :support_cas=>false, :tcp_nodelay=>false, :show_backtraces=>false, :retry_timeout=>30, :timeout=>1.0, :rcv_timeout=>1.0, :poll_timeout=>1.0, :connect_timeout=>4, :prefix_key=>"", :prefix_delimiter=>"", :hash_with_prefix_key=>true, :default_ttl=>604800, :default_weight=>8, :sort_hosts=>false, :auto_eject_hosts=>true, :server_failure_limit=>2, :verify_key=>true, :use_udp=>false, :binary_protocol=>false, :credentials=>nil, :experimental_features=>false, :exception_retry_limit=>5, :exceptions_to_retry=>[Memcached::ServerIsMarkedDead, Memcached::ATimeoutOccurred, Memcached::ConnectionBindFailure, Memcached::ConnectionFailure, Memcached::ConnectionSocketCreateFailure, Memcached::Failure, Memcached::MemoryAllocationFailure, Memcached::ReadFailure, Memcached::ServerEnd, Memcached::ServerError, Memcached::SystemError, Memcached::UnknownReadFailure, Memcached::WriteFailure, Memcached::SomeErrorsWereReported]}, @default_ttl=604800, @not_found=#<Memcached::NotFound: Memcached::NotFound>, @not_stored=#<Memcached::NotStored: Memcached::NotStored>>
irb(main):004:0>     mc.set "foo", 1, 100
=> nil
irb(main):005:0>     mc.get "foo"    
=> 1
irb(main):006:0>     pid1 = fork do
irb(main):007:1*     end
=> 9682
irb(main):008:0>     sleep 1
=> 1
irb(main):009:0>     mc.get "foo"    
Memcached::ServerIsMarkedDead: Key {"foo"=>"localhost:11211:8"}
    from /Users/me/.rvm/gems/ruby-1.9.2-p320@global/gems/memcached-1.5.0/lib/memcached/memcached.rb:630:in `reraise'
    from /Users/me/.rvm/gems/ruby-1.9.2-p320@global/gems/memcached-1.5.0/lib/memcached/memcached.rb:608:in `check_return_code'
    from /Users/me/.rvm/gems/ruby-1.9.2-p320@global/gems/memcached-1.5.0/lib/memcached/memcached.rb:517:in `get'
    from (irb):9
    from /Users/me/.rvm/rubies/ruby-1.9.2-p320/bin/irb:12:in `<main>'
irb(main):010:0>

And in 1.8.7

MBPR:$ rvm use 1.8.7
Using /Users/me/.rvm/gems/ruby-1.8.7-p370
MBPR:$ irb
1.8.7 :001 >     require 'rubygems'
 => true 
1.8.7 :002 >     require 'memcached'
 => true 
1.8.7 :003 >     mc = Memcached.new("localhost:11211", :timeout => 1.0)
 => #<Memcached:0x10fc65dd0 @not_stored=#<Memcached::NotStored: Memcached::NotStored>, @servers=["localhost:11211:8"], @not_found=#<Memcached::NotFound: Memcached::NotFound>, @default_ttl=604800, @struct=#<Rlibmemcached::MemcachedSt:0x10fc65808>, @options={:cache_lookups=>true, :retry_timeout=>30, :rcv_timeout=>1.0, :exception_retry_limit=>5, :show_backtraces=>false, :binary_protocol=>false, :hash_with_prefix_key=>true, :server_failure_limit=>2, :exceptions_to_retry=>[Memcached::ServerIsMarkedDead, Memcached::ATimeoutOccurred, Memcached::ConnectionBindFailure, Memcached::ConnectionFailure, Memcached::ConnectionSocketCreateFailure, Memcached::Failure, Memcached::MemoryAllocationFailure, Memcached::ReadFailure, Memcached::ServerError, Memcached::SystemError, Memcached::UnknownReadFailure, Memcached::WriteFailure], :no_block=>false, :distribution=>:consistent_ketama, :timeout=>1.0, :ketama_weighted=>true, :prefix_delimiter=>"", :auto_eject_hosts=>true, :support_cas=>false, :hash=>:fnv1_32, :buffer_requests=>false, :poll_timeout=>1.0, :default_ttl=>604800, :sort_hosts=>false, :verify_key=>true, :tcp_nodelay=>false, :connect_timeout=>4, :default_weight=>8, :credentials=>nil, :use_udp=>false}> 
1.8.7 :004 >     mc.set "foo", 1, 100
 => nil 
1.8.7 :005 >     mc.get "foo"    
 => 1 
1.8.7 :006 >     pid1 = fork do
1.8.7 :007 >         end
 => 9799 
1.8.7 :008 >     sleep 1
 => 1 
1.8.7 :009 >     mc.get "foo"    
 => 1 
1.8.7 :010 >
esilver
  • 27,713
  • 23
  • 122
  • 168
  • Are you using the same version of `memcached` in both cases? – davogones May 24 '13 at 05:06
  • Usually when a forked process terminates, it closes all filehandles, and this would include the `mc` Memcached handle. – tadman May 24 '13 at 05:26
  • @tadman you're right! Ultimately this code is running in passenger and nginx, and this situation is similar to 12.3.1 in http://www.modrails.com/documentation/Users%20guide%20Nginx.html But now Worker Process 2 will terminate the connection held by Worker Process 1 when it closes, even if I create a new connection at the start of Worker Process 2. That seems like wrong behavior. Any ideas how to correct? – esilver May 24 '13 at 05:46

1 Answers1

1

This is fundamental to the UNIX process model with respect to fork. Both programs are absolutely identical clones, and the only way to differentiate one from the other is the result of the fork call.

The clone will terminate any connections as the master would when the process completes. That this doesn't happen in one version of Ruby is probably a bug.

This is why forking before opening connections is probably a good idea, and forking after opening connections requires special handling.

tadman
  • 208,517
  • 23
  • 234
  • 262