3

I'm getting an exception in production which isn't providing and stacktrace information. How do I start debugging where this might be coming from?

Oct 25 16:26:17 socket-proxy app/web.1: Exception: RedisError: Disconnected (Redis::DisconnectedError) 
Oct 25 16:26:17 socket-proxy app/web.1: 0x4af6ac: ??? at ?? 
Oct 25 16:26:17 socket-proxy app/web.1: 0x4ce900: ??? at ?? 
Oct 25 16:26:17 socket-proxy app/web.1: 0x4b553e: ??? at ?? 
Oct 25 16:26:17 socket-proxy app/web.1: 0x529d1c: ??? at ?? 
Oct 25 16:26:17 socket-proxy app/web.1: 0x518cb2: ??? at ?? 
Oct 25 16:26:17 socket-proxy app/web.1: 0x518064: ??? at ?? 
Oct 25 16:26:17 socket-proxy app/web.1: 0x521d82: ??? at ?? 
Oct 25 16:26:17 socket-proxy app/web.1: 0x51ed3b: ??? at ?? 
Oct 25 16:26:17 socket-proxy app/web.1: 0x5240e9: ??? at ?? 
Oct 25 16:26:17 socket-proxy app/web.1: 0x50b995: ??? at ?? 
Oct 25 16:26:17 socket-proxy app/web.1: 0x416209: ??? at ?? 
Oct 25 16:26:17 socket-proxy app/web.1: 0x0: ??? at ?? 
Daniel Westendorf
  • 3,375
  • 18
  • 23
  • 2
    Are you compiling with `--no-debug`? – Stephie Oct 25 '17 at 23:39
  • @RX14 that was a good lead. I'm using the Heroku buildpack which seems to compile with `--no-debug`. I'll try w/o that flag and hopefully get a better stack trace. – Daniel Westendorf Oct 26 '17 at 14:13
  • I tried turning off of `--no-debug` with a PR to the heroku buildpack, but run into compile issues. https://github.com/crystal-lang/heroku-buildpack-crystal/pull/26 https://github.com/crystal-lang/crystal/issues/4719 – Daniel Westendorf Oct 26 '17 at 15:07
  • 1
    Yeah you should instead send a PR to attempt a non-release build, as --release currently requires --no-debug. Without release its still very fast so don't worry. – Stephie Oct 26 '17 at 18:08
  • I'll update the PR, thanks @RX14 – Daniel Westendorf Oct 26 '17 at 20:30

2 Answers2

4

Ruby dev here so not sure why stack trace is printed mysteriously, however if you are looking for some clues as to where to look I would start at this class:

redis/error.cr

# Exception for errors that Redis returns.
class Redis::Error < Exception
  def initialize(s)
    super("RedisError: #{s}")
  end
end

class Redis::DisconnectedError < Redis::Error
  def initialize
    super("Disconnected")
  end
end

Now clearly only place that exception seems to being raised in the crystal-redis repository is in this class:

redis/connection.cr (line: )

def receive_line
    line = @socket.gets(chomp: false)
    unless line
      raise Redis::DisconnectedError.new
    end
    line.byte_slice(0, line.bytesize - 2)
end

Looking at the method that uses it, receive_line it seems the error is clearly being thrown at Redis::Connection during connection or receive method.

So either its a error during connection, or a dropped connection.

Considering the clueless stack-trace, that would be a good start, unless you can share some more code to look at.

Hope that helps.

Shaunak
  • 17,377
  • 5
  • 53
  • 84
1

This ended up being because of the production server timing out the redis connection after a period of time. I've switched to redis-reconnect to auto-reconnect.

https://github.com/danielwestendorf/redis-reconnect

Daniel Westendorf
  • 3,375
  • 18
  • 23