Start Riak crashing after 30 seconds

Question

$ riak start crashing after 30 seconds of its start. I'm having following (changes) settings in my riak.conf:

search = on
storage_backend = leveldb
riak_control = on

crash.log contains the following:

2016-06-30 14:49:38 =ERROR REPORT====
** Generic server yz_solr_proc terminating 
** Last message in was {check_solr,0}
** When Server state == {state,"./data/yz",#Port<0.9441>,8093,8985}
** Reason for termination == 
** "solr didn't start in alloted time"
2016-06-30 14:49:38 =CRASH REPORT====
  crasher:
    initial call: yz_solr_proc:init/1
    pid: <0.582.0>
    registered_name: yz_solr_proc
    exception exit: {"solr didn't start in alloted time",[{gen_server,terminate,6,[{file,"gen_server.erl"},{line,744}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}
    ancestors: [yz_solr_sup,yz_sup,<0.578.0>]
    messages: [{'EXIT',#Port<0.9441>,normal}]
    links: [<0.580.0>]
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 376
    stack_size: 27
    reductions: 16170
  neighbours:
2016-06-30 14:49:38 =SUPERVISOR REPORT====
     Supervisor: {local,yz_solr_sup}
     Context:    child_terminated
     Reason:     "solr didn't start in alloted time"
     Offender:   [{pid,<0.582.0>},{name,yz_solr_proc},{mfargs,{yz_solr_proc,start_link,["./data/yz","./data/yz_temp",8093,8985]}},{restart_type,permanent},{shutdown,5000},{child_type,worker}]

2016-06-30 14:49:39 =ERROR REPORT====
** Generic server yz_solr_proc terminating 
** Last message in was {#Port<0.12204>,{exit_status,1}}
** When Server state == {state,"./data/yz",#Port<0.12204>,8093,8985}
** Reason for termination == 
** {"solr OS process exited",1}
2016-06-30 14:49:39 =CRASH REPORT====
  crasher:
    initial call: yz_solr_proc:init/1
    pid: <0.7631.0>
    registered_name: yz_solr_proc
    exception exit: {{"solr OS process exited",1},[{gen_server,terminate,6,[{file,"gen_server.erl"},{line,744}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}
    ancestors: [yz_solr_sup,yz_sup,<0.578.0>]
    messages: [{'EXIT',#Port<0.12204>,normal}]
    links: [<0.580.0>]
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 1598
    stack_size: 27
    reductions: 8968
  neighbours:
2016-06-30 14:49:39 =SUPERVISOR REPORT====
     Supervisor: {local,yz_solr_sup}
     Context:    child_terminated
     Reason:     {"solr OS process exited",1}
     Offender:   [{pid,<0.7631.0>},{name,yz_solr_proc},{mfargs,{yz_solr_proc,start_link,["./data/yz","./data/yz_temp",8093,8985]}},{restart_type,permanent},{shutdown,5000},{child_type,worker}]

2016-06-30 14:49:39 =SUPERVISOR REPORT====
     Supervisor: {local,yz_solr_sup}
     Context:    shutdown
     Reason:     reached_max_restart_intensity
     Offender:   [{pid,<0.7631.0>},{name,yz_solr_proc},{mfargs,{yz_solr_proc,start_link,["./data/yz","./data/yz_temp",8093,8985]}},{restart_type,permanent},{shutdown,5000},{child_type,worker}]

2016-06-30 14:49:39 =SUPERVISOR REPORT====
     Supervisor: {local,yz_sup}
     Context:    child_terminated
     Reason:     shutdown
     Offender:   [{pid,<0.580.0>},{name,yz_solr_sup},{mfargs,{yz_solr_sup,start_link,[]}},{restart_type,permanent},{shutdown,5000},{child_type,supervisor}]

2016-06-30 14:49:39 =SUPERVISOR REPORT====
     Supervisor: {local,yz_sup}
     Context:    shutdown
     Reason:     reached_max_restart_intensity
     Offender:   [{pid,<0.580.0>},{name,yz_solr_sup},{mfargs,{yz_solr_sup,start_link,[]}},{restart_type,permanent},{shutdown,5000},{child_type,supervisor}]

Which version of Riak are you using? Do you see anything in /var/log/riak/solr.log? — vempo, Jul 01 '16 at 06:48
Do you have a solr.log on your system? Check if for exceptions and error messages, it looks like the JVM crashes for some reason. — vempo, Jul 01 '16 at 18:25
solr.log 2016-07-01 12:13:33,697 [DEBUG] @HttpParser.java:281 filled 106/106 2016-07-01 12:13:33,697 [DEBUG] @Server.java:363 REQUEST /internal_solr/admin/cores on BlockingHttpConnection@2b406e05,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=-5,l=1,c=0},r=27682 2016-07-01 12:13:33,697 [DEBUG] @ContextHandler.java:931 scope null||/internal_solr/admin/cores @ — Masudur Rahman, Jul 01 '16 at 19:16
The solr.log seems clipped, and from a different date. It could help if you added it in full or at least its tail to the question. — vempo, Jul 02 '16 at 06:23
Finally fixed that issue. Cause /etc/hosts file did't have local ip mapping as follows(`/etc/hosts`): `127.0.0.1 localhost` This localhost and ip used solr as default host so after added that mapping into /etc/hosts file it resolved my issue. Thanks — Masudur Rahman, Jul 28 '16 at 21:34

score 2 · Answer 1 · answered Jul 02 '16 at 07:03

2

Make sure the ports used by Solr are available. The defaults are 8093 for search, and 8985 for JMX.
Tune your system to improve performance. Follow Improving Performance for Linux.
In riak.conf, increase the JVM's heap size, the default of 1G is often not enough. For example, search.solr.jvm_options=-d64 -Xms2g -Xmx4g -XX:+UseStringCache -XX:+UseCompressedOops (see Search Settings).
On a slow machine, Solr just may take longer to start. Try increasing search.solr.start_timeout.

Solr directories must be writable (usually /var/lib/riak/data/yz*), and a compatible JVM be used.

answered Jul 02 '16 at 07:03

vempo

3,093
1
14
16

Thanks, but that didn't helped me. – Masudur Rahman Jul 03 '16 at 18:58
@MasudurRahman post the trailing part of your solr.log please, search for ERROR and FATAL. – vempo Jul 04 '16 at 06:07
2016-07-04 22:48:43,518 [INFO] @LoggingInfoStream.java:34 [IW][qtp1340328248-54 - /internal_solr/admin/cores?action=STATUS&wt=json]: nrtIsCurrent: infoVersion matches: true; DW changes: false; BD changes: false 2016-07-04 22:48:43,518 [INFO] @LoggingInfoStream.java:34 [DW][qtp1340328248-54 - /internal_solr/admin/cores?action=STATUS&wt=json]: anyChanges? numDocsInRam=0 deletes=false hasTickets:false pendingChangesInFullFlush: false – Masudur Rahman Jul 05 '16 at 05:52
This chunk of solr.log doesn't help. There must be some `ERROR` or `FATAL` that prevents Solr from starting. BTW, how many Riak nodes, and with what `n_val` are you running? It is pretty common for people who just get started with Riak to run a single node without tuning the ring size and replication accordingly. Has your Riak ever started normally? Is there any data in it? – vempo Jul 27 '16 at 05:29
1

Anyway, finally fixed that issue. Cause file did't have local ip mapping as follows(`/etc/hosts`): `127.0.0.1 localhost` This localhost and ip used solr as default host so after added that mapping into /etc/hosts file it resolved my issue. Thanks – Masudur Rahman Jul 28 '16 at 07:11
Great! You might want to add this info as an answer or an update to your original question so that others could benefit from it. – vempo Jul 28 '16 at 08:13

score 0 · Answer 2 · answered Jul 28 '16 at 21:40

0

Riak's internal solr use localhost and 127.0.0.1 as default host. So it should have defined in /etc/hosts file:

127.0.0.1 localhost

FYI, if you use windows your hosts file location could be different.

answered Jul 28 '16 at 21:40

Masudur Rahman

86
2
6

Start Riak crashing after 30 seconds

2 Answers2