logstash unexpectedly stopped

Question

I have set up a logstash with redis architecture to handle my logs. The way I have organized it is:

logstash ---> redis ---> logstash ---> elasticsearch

but the problem that occurred is that after parsing nearly 1.25 million logs a java exception is thrown.

In my logstash.err log file, the exception appears as

Exception in thread "<file" java.lang.UnsupportedOperationException
    at java.lang.Thread.stop(Thread.java:869)
    at org.jruby.RubyThread.exceptionRaised(RubyThread.java:1221)
    at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:112)
    at java.lang.Thread.run(Thread.java:745)

I think that this exception might be thrown because of logstash unable to open/close a file. So what can I do to rectify this error? The way that my input configuration is set for my first logstash server to send the logs is:

input {
    file {
        start_position => "beginning"
        path => [
            "/var/logstash_logs/child1/nginx/*log*",
            "/var/logstash_logs/child2/nginx/*log*",
            "/var/logstash_logs/child3/nginx/*log*"
        ]
    }
}

And the way output is sent is like this:

output {
    redis {
        host => "X.X.X.X"
        key => "logstash"
        data_type => "list"
    }
}

There are no errors in the logs of logstash server with redis installed.

score 0 · Answer 1 · answered Aug 07 '15 at 08:09

0

Well, one problem here is with JRuby, which is trying to call Thread.stop(Throwable obj);, a deprecated method that throws UnsupportedOperationException and totally messes up the actual source of the error (the Throwable parameter).

So currently you can only guess what the actual problem is, and guessing is never good.

One idea is to set a breakpoint on RubyThread.exceptionRaised(); and run it through a debugger. That should allow you to find out what the original Throwable is, and then you can get to the source of the problem.

You should also check if there exists a bug ticket for JRuby about this, and possibly update your JRuby.

answered Aug 07 '15 at 08:09

Kayaman

72,141
5
83
121

Updated my JRuby and restarted the logstash. Hope that works fine. And about the `RubyThread.exceptionRaised();` breakpoint, I can't figure out how to set a breakpoint in my logstash config file. Can you please help me on how to do that? Thanks! :) – Keshav Agarwal Aug 07 '15 at 08:34
You can't set breakpoints in config files, but it would be quite a hassle to get a breakpoint inside JRuby. Go look around in the [issue tracker](https://github.com/jruby/jruby/issues) to see if they're aware of this bug. – Kayaman Aug 07 '15 at 08:36
Well, I submitted a new bug to them anyways. Seems the current version still has that bug. – Kayaman Aug 07 '15 at 08:50
Okay, hope they issue a solution soon. For me, the error is being thrown every 90 minutes when the files are being read. You can add that to the issue as well if it is of any relevance. – Keshav Agarwal Aug 07 '15 at 10:06
@KeshavAgarwal I doubt they will issue a solution any time soon (you might still want to follow the bug ticket to see what happens). If you're not very familiar with Java, it will probably be a lot of work to debug the cause at the moment. – Kayaman Aug 07 '15 at 10:11
1

I used kind of a hack to solve my case. I have set up a cron job for every hour to backup logstash's logs and restart it. And since the error occurs every 90 minutes, logstash doesn't come to halt. – Keshav Agarwal Aug 07 '15 at 10:30

logstash unexpectedly stopped

1 Answers1